Oct 31 00:26:54.701108 ip-10-0-134-252 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 00:26:54.701123 ip-10-0-134-252 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 00:26:54.701133 ip-10-0-134-252 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 00:26:54.701473 ip-10-0-134-252 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 00:27:04.887360 ip-10-0-134-252 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 00:27:04.887379 ip-10-0-134-252 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5fcc6b6002cc4d59ac0f95b702b9f0c5 -- Oct 31 00:29:05.849937 ip-10-0-134-252 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:29:06.216375 ip-10-0-134-252 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:29:06.216375 ip-10-0-134-252 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:29:06.216375 ip-10-0-134-252 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:29:06.216375 ip-10-0-134-252 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:29:06.219916 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219774 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219918 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219924 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219927 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219931 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219933 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219936 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219938 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219941 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219944 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219947 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219950 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219952 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219955 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219958 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219960 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219963 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:06.219958 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219966 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219969 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219972 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219983 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219986 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219989 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219991 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.219994 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220004 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220007 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220010 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220012 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220015 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220018 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220021 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220023 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220026 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220028 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220030 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220033 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:06.220358 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220036 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220038 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220042 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220044 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220046 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220049 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220052 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220054 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220057 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220059 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220061 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220064 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220067 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220069 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220073 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220075 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220078 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220080 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220083 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:06.220845 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220086 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220088 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220092 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220094 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220097 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220100 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220102 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220105 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220107 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220111 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220115 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220118 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220121 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220123 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220126 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220129 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220131 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220134 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220136 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:06.221324 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220138 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220142 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220145 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220147 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220150 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220152 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220155 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220157 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220161 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220163 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220166 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220168 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220171 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220173 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220176 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220178 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220181 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220183 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220186 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220188 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:06.221778 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220191 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220193 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220195 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220198 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220200 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220203 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220205 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220207 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220210 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220212 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.220215 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221012 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221019 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221022 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221026 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221029 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221031 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221035 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221037 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221040 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:06.222245 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221045 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221048 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221051 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221054 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221056 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221059 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221062 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221064 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221067 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221069 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221072 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221074 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221077 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221079 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221082 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221084 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221087 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221089 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221092 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221095 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:06.222731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221097 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221099 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221102 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221104 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221107 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221109 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221112 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221121 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221144 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221148 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221151 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221153 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221156 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221159 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221162 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221164 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221167 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221169 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221172 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221174 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:06.223207 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221177 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221179 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221182 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221184 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221187 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221189 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221192 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221194 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221198 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221202 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221205 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221208 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221211 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221214 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221218 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221222 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221225 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221227 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:06.223689 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221230 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221232 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221236 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221239 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221241 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221244 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221247 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221250 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221252 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221255 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221257 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221260 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221262 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221265 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221267 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221269 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221272 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221274 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221277 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:06.224164 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221280 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221282 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221284 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221287 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221290 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221293 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221295 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221297 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221300 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221302 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221304 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221307 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221309 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221312 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221314 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221317 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221320 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221322 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221325 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221328 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:06.224762 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221922 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221929 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221932 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221935 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221937 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221940 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221943 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221945 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221948 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221950 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221953 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221955 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221958 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221960 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221963 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221966 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221968 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221971 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221973 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:06.225235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221976 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221979 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221981 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221983 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221986 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221989 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221991 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221994 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221996 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.221999 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222002 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222005 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222009 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222011 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222014 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222017 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222020 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222022 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222025 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222027 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:06.225712 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222030 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222032 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222035 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222037 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222040 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222042 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222044 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222047 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222049 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222052 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222054 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222057 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222061 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222064 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222068 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222071 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222073 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222076 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222079 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:06.226184 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222081 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222085 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222089 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222093 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222096 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222099 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222102 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222105 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222108 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222111 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222113 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222116 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222118 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222121 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222123 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222125 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222129 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222132 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222135 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:06.226634 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222137 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222140 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222143 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222145 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222148 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222151 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222154 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222156 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222159 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222161 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222163 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222166 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222168 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222171 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222173 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222176 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222178 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222181 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222184 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222186 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:06.227102 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222189 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222191 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222194 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222196 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222199 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222201 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222204 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222206 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.222212 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223299 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223308 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223315 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223319 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223324 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223327 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223333 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223337 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223340 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223343 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223347 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223351 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223354 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:29:06.227568 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223357 2576 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223360 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223363 2576 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223365 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223368 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223373 2576 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223376 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223379 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223382 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223386 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223390 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223393 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223396 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223400 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223403 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223406 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223409 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223412 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223415 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223419 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223422 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223425 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223428 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223431 2576 flags.go:64] FLAG: --enable-server="true" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223433 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:29:06.228096 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223438 2576 flags.go:64] FLAG: --event-burst="100" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223441 2576 flags.go:64] FLAG: --event-qps="50" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223443 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223446 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223449 2576 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223453 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223456 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223459 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223462 2576 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223465 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223468 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223471 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223473 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223476 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223479 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223482 2576 flags.go:64] FLAG: --feature-gates="" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223487 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223490 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223493 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223496 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223499 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223502 2576 flags.go:64] FLAG: --help="false" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223505 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223508 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:29:06.228725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223511 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223514 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223518 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223521 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223524 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223527 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223530 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223532 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223535 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223538 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223541 2576 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223544 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223546 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223549 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223552 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223555 2576 flags.go:64] FLAG: --lock-file="" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223557 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223560 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223563 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223568 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223571 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223574 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223577 2576 flags.go:64] FLAG: --logging-format="text" Oct 31 00:29:06.229283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223580 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223583 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223586 2576 flags.go:64] FLAG: --manifest-url="" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223589 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223594 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223597 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223601 2576 flags.go:64] FLAG: --max-pods="110" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223604 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223607 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223610 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223613 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223628 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223632 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223635 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223643 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223645 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223649 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223651 2576 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223654 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223657 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223660 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223663 2576 flags.go:64] FLAG: --port="10250" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223666 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223669 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0684ee449620158ad" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223672 2576 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223675 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:29:06.229834 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223677 2576 flags.go:64] FLAG: --register-node="true" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223680 2576 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223684 2576 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223686 2576 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223689 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223692 2576 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223697 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223700 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223703 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223707 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223710 2576 flags.go:64] FLAG: --runonce="false" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223713 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223716 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223719 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223722 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223724 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223728 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223731 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223734 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223737 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223739 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223743 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223745 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223748 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223751 2576 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:29:06.230677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223754 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223759 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223762 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223765 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223769 2576 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223772 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223775 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223778 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223780 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223783 2576 flags.go:64] FLAG: --v="2" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223787 2576 flags.go:64] FLAG: --version="false" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223792 2576 flags.go:64] FLAG: --vmodule="" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223795 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.223798 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.223954 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.223958 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.223961 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.223964 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.223970 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.223980 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.223985 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.223987 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:06.231723 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.223990 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.223994 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.223997 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224000 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224003 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224005 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224008 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224010 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224013 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224015 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224018 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224020 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224023 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224026 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224028 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224031 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224033 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224036 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224038 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224041 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:06.232684 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224043 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224046 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224048 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224050 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224053 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224056 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224058 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224061 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224065 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224068 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224070 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224073 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224075 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224079 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224081 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224084 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224086 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224089 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224091 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224094 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:06.233354 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224096 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224099 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224101 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224104 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224106 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224109 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224111 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224114 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224117 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224119 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224122 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224124 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224127 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224129 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224133 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224137 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224140 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224142 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224145 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:06.233862 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224147 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224152 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224156 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224159 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224161 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224163 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224169 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224171 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224174 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224177 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224180 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224182 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224185 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224187 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224190 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224192 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224195 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224197 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224200 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:06.234751 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224203 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224205 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224208 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224211 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224213 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224216 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224218 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224221 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224223 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224226 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224228 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224231 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224233 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224236 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224239 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224242 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224245 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224247 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224250 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:06.235581 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224253 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.224280 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224317 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224323 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224326 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224328 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224331 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224334 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224336 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224339 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224342 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224344 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224347 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224349 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224352 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224355 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:06.236350 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224357 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224359 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224362 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224364 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224367 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224370 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224372 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224374 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224377 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224379 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224382 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224385 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224388 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224390 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224393 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224396 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224398 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224402 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224405 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224407 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:06.236766 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224410 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224413 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224415 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224418 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224420 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224423 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224425 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224428 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224430 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224433 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224436 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224438 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224441 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224445 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224449 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224453 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224456 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224459 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224462 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:06.237541 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224465 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224469 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224472 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224475 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224479 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224481 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224484 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224486 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224489 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224492 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224495 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224498 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224500 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224503 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224505 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224508 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224511 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224513 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224516 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:06.238356 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224518 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224521 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224524 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224526 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224529 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224531 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224534 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224536 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224538 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224541 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224543 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224546 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224548 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224550 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224553 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224555 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224557 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224562 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224564 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:06.239003 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224567 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224569 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224572 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224575 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224578 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224581 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224583 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224586 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224588 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224591 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224593 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224596 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224599 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224602 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.224604 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.233838 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.233857 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.233878 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.233885 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:29:06.239609 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234003 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234012 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234017 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234023 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234027 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234031 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234036 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234040 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234044 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234048 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234052 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234056 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234060 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234065 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234070 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234074 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234087 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234091 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234095 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234100 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:06.240098 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234104 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234108 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234112 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234117 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234121 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234128 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234134 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234138 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234143 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234149 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234153 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234158 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234162 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234167 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234171 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234176 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234180 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234184 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234189 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:06.240643 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234193 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234197 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234201 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234205 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234210 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234214 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234218 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234222 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234226 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234230 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234242 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234246 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234258 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234262 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234265 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234272 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234279 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234284 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234288 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:06.241089 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234292 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234297 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234301 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234306 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234310 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234314 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234319 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234323 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234328 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234332 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234336 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234340 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234344 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234348 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234352 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234356 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234360 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234364 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234369 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:06.241650 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234373 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234377 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234381 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234385 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234389 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234401 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234406 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234410 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234415 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234419 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234423 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234427 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234431 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234435 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234440 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234444 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234449 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234453 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234457 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234462 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:06.242101 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234466 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234470 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234474 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234478 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234482 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234486 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234490 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234494 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234498 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.234540 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234897 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234908 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234913 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234919 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234923 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234928 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:06.242947 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234932 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234936 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234948 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234953 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234958 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234962 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234967 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234971 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234975 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234980 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234985 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234989 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234993 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.234998 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235002 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235006 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235013 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235018 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235024 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235028 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:06.243576 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235032 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235036 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235041 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235045 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235049 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235053 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235057 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235061 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235065 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235070 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235074 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235078 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235082 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235086 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235090 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235102 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235106 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235110 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235114 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235118 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:06.244235 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235123 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235127 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235132 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235136 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235140 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235144 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235148 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235153 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235157 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235162 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235166 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235170 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235174 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235179 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235183 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235187 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235191 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235195 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235199 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235203 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:06.244880 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235207 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235211 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235215 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235220 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235224 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235228 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235232 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235236 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235247 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235252 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235256 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235260 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235264 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235269 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235273 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235278 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235282 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235286 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235296 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235301 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:06.245582 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235305 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235309 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235313 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235317 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235321 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235326 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235330 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235334 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235338 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235345 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235350 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235355 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235359 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235363 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235367 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235371 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235375 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235380 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235384 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:06.246288 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235388 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.235427 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235501 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235508 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235514 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235519 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235523 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235528 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235533 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235537 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235541 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235545 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235549 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235553 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235557 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235562 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:06.246930 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235566 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235570 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235574 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235579 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235583 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235586 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235590 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235594 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235598 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235602 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235606 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235610 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235614 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235638 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235645 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235649 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235654 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235658 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235662 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235675 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:06.247319 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235680 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235684 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235688 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235692 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235697 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235701 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235705 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235709 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235714 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235718 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235722 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235726 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235730 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235734 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235739 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235743 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235747 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235752 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235756 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:06.247860 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235760 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235764 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235769 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235773 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235777 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235781 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235786 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235790 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235794 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235807 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235812 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235816 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235820 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235831 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235836 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235840 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235844 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235848 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235852 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235857 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:06.248333 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235861 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235865 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235869 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235873 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235877 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235881 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235885 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235889 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235894 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235898 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235905 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235910 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235914 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235918 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235922 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235926 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235930 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235935 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235939 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235943 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:06.248848 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235947 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235951 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235955 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235960 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235964 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235968 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.235994 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.236003 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.236009 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.236015 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.236019 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.236024 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:06.236028 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.236108 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.239205 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.240376 2576 server.go:1014] "Starting client certificate rotation" Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.240482 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:29:06.249309 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.240532 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:29:06.249756 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.242995 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:29:06.251153 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.251134 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:29:06.262239 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.262221 2576 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:29:06.266046 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.266032 2576 log.go:25] "Validated CRI v1 image API" Oct 31 00:29:06.267027 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.266985 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:29:06.271634 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.271597 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a9400b02-3866-42d7-8b09-596edb3239a5:/dev/nvme0n1p4 acdf1e7a-0579-4ba1-bd25-8939952ea356:/dev/nvme0n1p3] Oct 31 00:29:06.271711 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.271631 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:29:06.273914 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.273897 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:29:06.276782 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.276668 2576 manager.go:217] Machine: {Timestamp:2026-10-31 00:29:06.275269636 +0000 UTC m=+0.328241598 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101662 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23d3eb8aa50bad25f7e995d2cbc58c SystemUUID:ec23d3eb-8aa5-0bad-25f7-e995d2cbc58c BootID:5fcc6b60-02cc-4d59-ac0f-95b702b9f0c5 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:df:da:f0:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:df:da:f0:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:89:a6:bb:6a:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:29:06.276782 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.276780 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:29:06.276924 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.276908 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:29:06.279647 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.279608 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:29:06.279816 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.279649 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-252.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:29:06.279864 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.279819 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:29:06.279864 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.279827 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:29:06.279864 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.279851 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:29:06.279941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.279867 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:29:06.280926 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.280915 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:29:06.285493 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.285481 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:29:06.285645 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.285635 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:29:06.285685 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.285668 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:29:06.285685 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.285679 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:29:06.285739 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.285703 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:29:06.285767 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.285759 2576 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:29:06.285794 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.285779 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:29:06.286315 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.286294 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:29:06.288889 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.288858 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:29:06.290269 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.290249 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wwc7g" Oct 31 00:29:06.297873 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.297718 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:29:06.298310 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.298288 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wwc7g" Oct 31 00:29:06.299434 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.299417 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:29:06.299494 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.299459 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:29:06.299741 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.299724 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:29:06.299814 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.299755 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:29:06.299814 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.299772 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:29:06.299814 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.299785 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:29:06.299904 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.299859 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:29:06.299904 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.299879 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:29:06.300718 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.300701 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:29:06.300718 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.300717 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:29:06.300972 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.300727 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:29:06.300972 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.300733 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:29:06.300972 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.300742 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:29:06.300972 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.300752 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:29:06.301666 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.301656 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:29:06.301666 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.301667 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:29:06.305940 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.305925 2576 server.go:1263] "Started kubelet" Oct 31 00:29:06.306026 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.306000 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:29:06.306133 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.306070 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:29:06.306187 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.306155 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:29:06.306683 ip-10-0-134-252 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:29:06.308225 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.308208 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:29:06.308447 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.308423 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:29:06.311883 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.311864 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-134-252.us-east-2.compute.internal" not found Oct 31 00:29:06.312007 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.311993 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:29:06.312278 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.312263 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:29:06.313204 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:06.313182 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:29:06.314779 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.314764 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:29:06.315432 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.315417 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:29:06.315525 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:06.315489 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-252.us-east-2.compute.internal\" not found" Oct 31 00:29:06.316146 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.316130 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:29:06.316212 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.316152 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:29:06.316212 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.316129 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:29:06.316300 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.316270 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:29:06.316300 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.316280 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:29:06.316300 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.316294 2576 factory.go:55] Registering systemd factory Oct 31 00:29:06.316402 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.316307 2576 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:29:06.316692 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.316676 2576 factory.go:153] Registering CRI-O factory Oct 31 00:29:06.316692 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.316692 2576 factory.go:223] Registration of the crio container factory successfully Oct 31 00:29:06.316801 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.316741 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:29:06.316801 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.316770 2576 factory.go:103] Registering Raw factory Oct 31 00:29:06.316801 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.316796 2576 manager.go:1199] Started watching for new ooms in manager Oct 31 00:29:06.318088 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.318074 2576 manager.go:319] Starting recovery of all containers Oct 31 00:29:06.319357 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.319339 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:29:06.321729 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:06.321699 2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-252.us-east-2.compute.internal\" not found" node="ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.325925 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.325897 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-134-252.us-east-2.compute.internal" not found Oct 31 00:29:06.329578 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.329562 2576 manager.go:324] Recovery completed Oct 31 00:29:06.333312 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.333299 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:29:06.335110 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.335091 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:29:06.335179 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.335126 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:29:06.335179 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.335136 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:29:06.335651 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.335605 2576 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:29:06.335651 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.335613 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:29:06.335651 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.335632 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:29:06.335771 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.335654 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:29:06.336337 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.336320 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:29:06.336400 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.336347 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:29:06.336400 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.336361 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:29:06.337735 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.337723 2576 policy_none.go:50] "Start" Oct 31 00:29:06.337769 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.337740 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:29:06.337769 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.337750 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:29:06.339569 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.339559 2576 policy_none.go:44] "Start" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.379177 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.382088 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-134-252.us-east-2.compute.internal" not found Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.383388 2576 manager.go:341] "Starting Device Plugin manager" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:06.383419 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.383431 2576 server.go:87] "Starting device plugin registration server" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.383691 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.383703 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.383805 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.383892 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.383899 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:06.384482 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:06.384517 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-252.us-east-2.compute.internal\" not found" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.389273 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.390497 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.391177 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.391217 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:06.391258 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 00:29:06.398435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.394816 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:29:06.484269 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.484218 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:29:06.485111 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.485096 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:29:06.485197 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.485127 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:29:06.485197 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.485144 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:29:06.485197 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.485173 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.491730 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.491713 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal"] Oct 31 00:29:06.491799 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.491766 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:29:06.492496 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.492481 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:29:06.492574 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.492512 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:29:06.492574 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.492526 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:29:06.493935 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.493917 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.493995 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:06.493941 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-252.us-east-2.compute.internal\": node \"ip-10-0-134-252.us-east-2.compute.internal\" not found" Oct 31 00:29:06.494867 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.494851 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.496722 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.496706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.516035 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.516018 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.529101 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.529082 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:29:06.529204 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.529110 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.529639 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.529600 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.531714 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.531700 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.546496 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.546477 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:29:06.546549 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.546540 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:29:06.546601 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:06.546581 2576 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.546653 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.546642 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:29:06.546692 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:06.546673 2576 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.618128 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.618104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/703d740fe3de84816febee5a3376c36a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"703d740fe3de84816febee5a3376c36a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.618205 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.618131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8f3794bd171ae09db1b2f1a9b17e1c85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"8f3794bd171ae09db1b2f1a9b17e1c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.618205 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.618148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f3794bd171ae09db1b2f1a9b17e1c85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"8f3794bd171ae09db1b2f1a9b17e1c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.718807 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.718779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/703d740fe3de84816febee5a3376c36a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"703d740fe3de84816febee5a3376c36a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.718807 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.718807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8f3794bd171ae09db1b2f1a9b17e1c85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"8f3794bd171ae09db1b2f1a9b17e1c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.718967 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.718824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f3794bd171ae09db1b2f1a9b17e1c85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"8f3794bd171ae09db1b2f1a9b17e1c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.718967 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.718872 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f3794bd171ae09db1b2f1a9b17e1c85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"8f3794bd171ae09db1b2f1a9b17e1c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.718967 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.718881 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/703d740fe3de84816febee5a3376c36a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"703d740fe3de84816febee5a3376c36a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.718967 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.718881 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8f3794bd171ae09db1b2f1a9b17e1c85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"8f3794bd171ae09db1b2f1a9b17e1c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.852507 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.852428 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:06.852661 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:06.852427 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:29:07.240359 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.240277 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:29:07.240875 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.240465 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:29:07.240875 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.240464 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:29:07.286660 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.286631 2576 apiserver.go:52] "Watching apiserver" Oct 31 00:29:07.294359 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.294334 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:29:07.294604 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.294589 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-swvwl","kube-system/global-pull-secret-syncer-wj84l","kube-system/konnectivity-agent-d4jrp","openshift-multus/multus-9zlgx","openshift-multus/multus-additional-cni-plugins-jqkmw","openshift-network-diagnostics/network-check-target-nsk8c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c","openshift-dns/node-resolver-m8d27","kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-pptjg","openshift-image-registry/node-ca-bmn4m","openshift-multus/network-metrics-daemon-87jmt","openshift-network-operator/iptables-alerter-9l4x8"] Oct 31 00:29:07.297165 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.297149 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.299235 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.299217 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:07.299317 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.299278 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:07.301185 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.301166 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:29:07.301324 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.301307 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:29:07.301492 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.301469 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:29:07.301830 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.301812 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:29:07.301892 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.301814 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:29:07.301892 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.301846 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:29:07.301892 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.301858 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:29:07.301892 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.301865 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pl6tx\"" Oct 31 00:29:07.303431 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.303412 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.303802 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.303770 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:24:06 +0000 UTC" deadline="2028-05-05 13:12:35.174672423 +0000 UTC" Oct 31 00:29:07.303802 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.303800 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13260h43m27.870874121s" Oct 31 00:29:07.305542 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.305528 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.307570 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.307554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:07.307675 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.307614 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:07.309765 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.309748 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.312035 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.311992 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m8d27" Oct 31 00:29:07.314417 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.314396 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.315019 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.315004 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:29:07.316813 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.316798 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:29:07.319585 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.319561 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:07.319693 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.319670 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:07.320494 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-registration-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.320591 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-sys\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.320591 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-kubelet\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.320688 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.320688 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/089a2aaa-c90a-4df7-8e97-09650f16aab4-agent-certs\") pod \"konnectivity-agent-d4jrp\" (UID: \"089a2aaa-c90a-4df7-8e97-09650f16aab4\") " pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:29:07.320747 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320711 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.320781 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkkh4\" (UniqueName: \"kubernetes.io/projected/f4b96af0-063c-4752-a290-12371f09d52a-kube-api-access-zkkh4\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:07.320781 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-systemd-units\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.320835 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-env-overrides\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.320835 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/089a2aaa-c90a-4df7-8e97-09650f16aab4-konnectivity-ca\") pod \"konnectivity-agent-d4jrp\" (UID: \"089a2aaa-c90a-4df7-8e97-09650f16aab4\") " pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:29:07.320891 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb2daade-8f58-42e8-a79b-2a373ff186ea-host\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:29:07.320927 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-netns\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.320958 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.320941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysctl-conf\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.321035 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-run-netns\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.321090 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.321136 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovnkube-script-lib\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.321182 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-cnibin\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.321211 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.321259 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2x6\" (UniqueName: \"kubernetes.io/projected/0b02c695-144a-4481-bc14-1cd89baba762-kube-api-access-bf2x6\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:29:07.321295 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpqqj\" (UniqueName: \"kubernetes.io/projected/db73288c-815d-49ed-9cdb-6799fb6ecc7b-kube-api-access-bpqqj\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.321339 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:07.321381 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-system-cni-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.321381 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-kubelet\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.321523 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-multus-certs\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.321523 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-log-socket\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.321632 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-kubelet-config\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:07.321689 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-hostroot\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.321689 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-run\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.321772 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33c29ec2-dcd1-4c23-99cc-66c6684db03a-tmp\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.321772 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-run-ovn-kubernetes\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.321860 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-system-cni-dir\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.321860 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-socket-dir-parent\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.321860 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-cni-multus\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.321977 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-etc-selinux\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.321977 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87hzr\" (UniqueName: \"kubernetes.io/projected/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-kube-api-access-87hzr\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.322075 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.321968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-slash\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.322075 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-etc-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.322075 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-ovn\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.322170 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-cni-netd\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.322170 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322093 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-daemon-config\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.322170 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-sys-fs\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.322170 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.322329 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkk68\" (UniqueName: \"kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68\") pod \"network-check-target-nsk8c\" (UID: \"698ab510-f0c2-4745-b410-b1697d1aabb8\") " pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:07.322329 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb2daade-8f58-42e8-a79b-2a373ff186ea-serviceca\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:29:07.322329 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-k8s-cni-cncf-io\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.322329 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-socket-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.322329 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysconfig\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.322329 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-node-log\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.322329 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-systemd\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.322329 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-var-lib-kubelet\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.322329 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322316 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-tuned\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcgv9\" (UniqueName: \"kubernetes.io/projected/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-kube-api-access-jcgv9\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-device-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vszj2\" (UniqueName: \"kubernetes.io/projected/fb2daade-8f58-42e8-a79b-2a373ff186ea-kube-api-access-vszj2\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-kubernetes\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysctl-d\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-cni-bin\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-cni-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-modprobe-d\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-lib-modules\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-var-lib-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322657 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b02c695-144a-4481-bc14-1cd89baba762-hosts-file\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-conf-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.322742 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgndg\" (UniqueName: \"kubernetes.io/projected/50ef0d54-5212-427c-8bab-04b4262c3e11-kube-api-access-xgndg\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322791 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-cnibin\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-cni-bin\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-host\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322882 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4967k\" (UniqueName: \"kubernetes.io/projected/33c29ec2-dcd1-4c23-99cc-66c6684db03a-kube-api-access-4967k\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-systemd\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.322973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b02c695-144a-4481-bc14-1cd89baba762-tmp-dir\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.323010 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db73288c-815d-49ed-9cdb-6799fb6ecc7b-cni-binary-copy\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.323034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-etc-kubernetes\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.323062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovnkube-config\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.323106 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovn-node-metrics-cert\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.323338 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.323133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-dbus\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:07.332283 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.332267 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:29:07.401975 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.401949 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:29:07.402265 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402246 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:29:07.402364 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402284 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:29:07.402364 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402297 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:29:07.402505 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402485 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:29:07.402602 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402510 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9nqcj\"" Oct 31 00:29:07.402602 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402577 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:29:07.402725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402614 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-l9cjs\"" Oct 31 00:29:07.402725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402639 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-72tmz\"" Oct 31 00:29:07.402725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402701 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:29:07.402725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402702 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:29:07.402946 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402929 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:29:07.403043 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402934 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:29:07.403043 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402984 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:29:07.403043 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.402987 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:29:07.403043 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403007 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:29:07.403043 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403013 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:29:07.403043 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403010 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:29:07.403043 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403026 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:29:07.403248 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403079 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:29:07.403316 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403304 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:29:07.403379 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403351 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jhx4l\"" Oct 31 00:29:07.403422 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403380 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:29:07.403481 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403466 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:29:07.403532 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403474 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rn9j5\"" Oct 31 00:29:07.403532 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403503 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:29:07.403532 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403514 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vxzr5\"" Oct 31 00:29:07.403532 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403505 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4cbmj\"" Oct 31 00:29:07.403672 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.403515 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pnvd5\"" Oct 31 00:29:07.417055 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.417037 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:29:07.424160 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpqqj\" (UniqueName: \"kubernetes.io/projected/db73288c-815d-49ed-9cdb-6799fb6ecc7b-kube-api-access-bpqqj\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424239 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:07.424239 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-system-cni-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424239 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-kubelet\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424239 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-multus-certs\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-log-socket\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-kubelet-config\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.424290 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-hostroot\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424296 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-system-cni-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424313 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-multus-certs\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-run\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424332 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-log-socket\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424291 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-kubelet\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-hostroot\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424368 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-kubelet-config\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.424385 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs podName:f4b96af0-063c-4752-a290-12371f09d52a nodeName:}" failed. No retries permitted until 2026-10-31 00:29:07.92434059 +0000 UTC m=+1.977312544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs") pod "network-metrics-daemon-87jmt" (UID: "f4b96af0-063c-4752-a290-12371f09d52a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:07.424418 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33c29ec2-dcd1-4c23-99cc-66c6684db03a-tmp\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424426 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-run-ovn-kubernetes\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424456 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-run\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424471 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-run-ovn-kubernetes\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-system-cni-dir\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-socket-dir-parent\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-cni-multus\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424569 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-system-cni-dir\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-etc-selinux\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424616 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-socket-dir-parent\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87hzr\" (UniqueName: \"kubernetes.io/projected/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-kube-api-access-87hzr\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424653 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-cni-multus\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-slash\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-etc-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-ovn\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424721 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-etc-selinux\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-cni-netd\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.424941 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424737 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-etc-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424718 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424752 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-slash\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-daemon-config\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424757 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-ovn\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424774 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-cni-netd\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-sys-fs\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkk68\" (UniqueName: \"kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68\") pod \"network-check-target-nsk8c\" (UID: \"698ab510-f0c2-4745-b410-b1697d1aabb8\") " pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424848 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-sys-fs\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/658d1734-2e40-4784-8c85-d57ce4c5c7a8-iptables-alerter-script\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/658d1734-2e40-4784-8c85-d57ce4c5c7a8-host-slash\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb2daade-8f58-42e8-a79b-2a373ff186ea-serviceca\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-k8s-cni-cncf-io\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.424996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-socket-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysconfig\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425063 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-k8s-cni-cncf-io\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.425790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-node-log\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-systemd\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-var-lib-kubelet\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425139 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysconfig\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-tuned\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgv9\" (UniqueName: \"kubernetes.io/projected/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-kube-api-access-jcgv9\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425177 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-socket-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425196 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-systemd\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-device-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425223 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-node-log\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vszj2\" (UniqueName: \"kubernetes.io/projected/fb2daade-8f58-42e8-a79b-2a373ff186ea-kube-api-access-vszj2\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425243 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-device-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-kubernetes\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425281 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysctl-d\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-cni-bin\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425350 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-daemon-config\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.426557 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-cni-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425353 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425438 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb2daade-8f58-42e8-a79b-2a373ff186ea-serviceca\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425477 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-var-lib-kubelet\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-modprobe-d\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-lib-modules\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-var-lib-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b02c695-144a-4481-bc14-1cd89baba762-hosts-file\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-conf-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425607 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysctl-d\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425684 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-lib-modules\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgndg\" (UniqueName: \"kubernetes.io/projected/50ef0d54-5212-427c-8bab-04b4262c3e11-kube-api-access-xgndg\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425714 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-cni-bin\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425722 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lndm5\" (UniqueName: \"kubernetes.io/projected/658d1734-2e40-4784-8c85-d57ce4c5c7a8-kube-api-access-lndm5\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425687 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-kubernetes\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425809 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-cnibin\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.427356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-cnibin\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-var-lib-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425823 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-modprobe-d\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-cni-bin\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425855 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-cni-bin\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.425765 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425890 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-conf-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425902 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-host\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425920 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b02c695-144a-4481-bc14-1cd89baba762-hosts-file\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.425926 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret podName:31d4d8b4-ee1c-4703-8ac5-f739d72ab13d nodeName:}" failed. No retries permitted until 2026-10-31 00:29:07.925907775 +0000 UTC m=+1.978879732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret") pod "global-pull-secret-syncer-wj84l" (UID: "31d4d8b4-ee1c-4703-8ac5-f739d72ab13d") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.425864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-host\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426033 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-cni-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426037 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4967k\" (UniqueName: \"kubernetes.io/projected/33c29ec2-dcd1-4c23-99cc-66c6684db03a-kube-api-access-4967k\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-systemd\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b02c695-144a-4481-bc14-1cd89baba762-tmp-dir\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426142 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-systemd\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.427994 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db73288c-815d-49ed-9cdb-6799fb6ecc7b-cni-binary-copy\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-etc-kubernetes\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovnkube-config\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovn-node-metrics-cert\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-dbus\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-registration-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-sys\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-kubelet\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/089a2aaa-c90a-4df7-8e97-09650f16aab4-agent-certs\") pod \"konnectivity-agent-d4jrp\" (UID: \"089a2aaa-c90a-4df7-8e97-09650f16aab4\") " pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkh4\" (UniqueName: \"kubernetes.io/projected/f4b96af0-063c-4752-a290-12371f09d52a-kube-api-access-zkkh4\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426544 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b02c695-144a-4481-bc14-1cd89baba762-tmp-dir\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426599 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-etc-kubernetes\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426650 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-systemd-units\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-systemd-units\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-env-overrides\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.428558 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426703 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db73288c-815d-49ed-9cdb-6799fb6ecc7b-cni-binary-copy\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426714 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/089a2aaa-c90a-4df7-8e97-09650f16aab4-konnectivity-ca\") pod \"konnectivity-agent-d4jrp\" (UID: \"089a2aaa-c90a-4df7-8e97-09650f16aab4\") " pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb2daade-8f58-42e8-a79b-2a373ff186ea-host\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-netns\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysctl-conf\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-run-netns\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovnkube-script-lib\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-cnibin\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2x6\" (UniqueName: \"kubernetes.io/projected/0b02c695-144a-4481-bc14-1cd89baba762-kube-api-access-bf2x6\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.427001 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-sys\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.427071 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-dbus\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.427120 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-kubelet\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.427187 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.427224 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-run-netns\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.429294 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.427277 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb2daade-8f58-42e8-a79b-2a373ff186ea-host\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.427315 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-netns\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.427318 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-registration-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.426275 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.427386 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.427413 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysctl-conf\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.427433 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-cnibin\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.427467 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.428101 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-env-overrides\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.428579 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovnkube-config\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.428583 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/089a2aaa-c90a-4df7-8e97-09650f16aab4-konnectivity-ca\") pod \"konnectivity-agent-d4jrp\" (UID: \"089a2aaa-c90a-4df7-8e97-09650f16aab4\") " pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.428592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.429155 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovnkube-script-lib\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.429308 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33c29ec2-dcd1-4c23-99cc-66c6684db03a-tmp\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.429960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.429557 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-tuned\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.430538 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.430518 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/089a2aaa-c90a-4df7-8e97-09650f16aab4-agent-certs\") pod \"konnectivity-agent-d4jrp\" (UID: \"089a2aaa-c90a-4df7-8e97-09650f16aab4\") " pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:29:07.430765 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.430746 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovn-node-metrics-cert\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.433906 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.433890 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:29:07.433906 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.433907 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:29:07.434012 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.433916 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jkk68 for pod openshift-network-diagnostics/network-check-target-nsk8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:07.434012 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.433955 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68 podName:698ab510-f0c2-4745-b410-b1697d1aabb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:07.933945048 +0000 UTC m=+1.986916998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jkk68" (UniqueName: "kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68") pod "network-check-target-nsk8c" (UID: "698ab510-f0c2-4745-b410-b1697d1aabb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:07.437493 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.436905 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpqqj\" (UniqueName: \"kubernetes.io/projected/db73288c-815d-49ed-9cdb-6799fb6ecc7b-kube-api-access-bpqqj\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.437493 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.437213 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87hzr\" (UniqueName: \"kubernetes.io/projected/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-kube-api-access-87hzr\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.438347 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.438322 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgndg\" (UniqueName: \"kubernetes.io/projected/50ef0d54-5212-427c-8bab-04b4262c3e11-kube-api-access-xgndg\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.438425 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.438368 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkh4\" (UniqueName: \"kubernetes.io/projected/f4b96af0-063c-4752-a290-12371f09d52a-kube-api-access-zkkh4\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:07.438666 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.438610 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgv9\" (UniqueName: \"kubernetes.io/projected/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-kube-api-access-jcgv9\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.438741 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.438684 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vszj2\" (UniqueName: \"kubernetes.io/projected/fb2daade-8f58-42e8-a79b-2a373ff186ea-kube-api-access-vszj2\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:29:07.438821 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.438800 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2x6\" (UniqueName: \"kubernetes.io/projected/0b02c695-144a-4481-bc14-1cd89baba762-kube-api-access-bf2x6\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:29:07.439402 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.439386 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4967k\" (UniqueName: \"kubernetes.io/projected/33c29ec2-dcd1-4c23-99cc-66c6684db03a-kube-api-access-4967k\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.528347 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.528268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/658d1734-2e40-4784-8c85-d57ce4c5c7a8-iptables-alerter-script\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:29:07.528347 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.528312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/658d1734-2e40-4784-8c85-d57ce4c5c7a8-host-slash\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:29:07.528709 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.528390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndm5\" (UniqueName: \"kubernetes.io/projected/658d1734-2e40-4784-8c85-d57ce4c5c7a8-kube-api-access-lndm5\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:29:07.528709 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.528407 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/658d1734-2e40-4784-8c85-d57ce4c5c7a8-host-slash\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:29:07.529697 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.529670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/658d1734-2e40-4784-8c85-d57ce4c5c7a8-iptables-alerter-script\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:29:07.537790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.537763 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lndm5\" (UniqueName: \"kubernetes.io/projected/658d1734-2e40-4784-8c85-d57ce4c5c7a8-kube-api-access-lndm5\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:29:07.630631 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:07.630589 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3794bd171ae09db1b2f1a9b17e1c85.slice/crio-9d4b6c1858c21c1556fccb4d0402efd73216d33f91e8ddfb133d21de4cf52d0c WatchSource:0}: Error finding container 9d4b6c1858c21c1556fccb4d0402efd73216d33f91e8ddfb133d21de4cf52d0c: Status 404 returned error can't find the container with id 9d4b6c1858c21c1556fccb4d0402efd73216d33f91e8ddfb133d21de4cf52d0c Oct 31 00:29:07.631172 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:07.631158 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703d740fe3de84816febee5a3376c36a.slice/crio-fd51a42ccb70e635a0e592c6baa68e6cb37b6bec9723874747a07385d7d40bf3 WatchSource:0}: Error finding container fd51a42ccb70e635a0e592c6baa68e6cb37b6bec9723874747a07385d7d40bf3: Status 404 returned error can't find the container with id fd51a42ccb70e635a0e592c6baa68e6cb37b6bec9723874747a07385d7d40bf3 Oct 31 00:29:07.632304 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.632286 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:07.635705 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.635688 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:29:07.641021 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:07.640992 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912d5cf1_39a7_4fe3_8e72_d6b20bd67476.slice/crio-583b9b18ba6ebc6dad4e716877a8995f997a662b6214e4cbabdfa9e9744447ca WatchSource:0}: Error finding container 583b9b18ba6ebc6dad4e716877a8995f997a662b6214e4cbabdfa9e9744447ca: Status 404 returned error can't find the container with id 583b9b18ba6ebc6dad4e716877a8995f997a662b6214e4cbabdfa9e9744447ca Oct 31 00:29:07.641579 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.641561 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:29:07.649312 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:07.649293 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089a2aaa_c90a_4df7_8e97_09650f16aab4.slice/crio-d0a3acbc83a5f38728133e3dfa5b0ea2507f076b332b8da29358f27740a86c2f WatchSource:0}: Error finding container d0a3acbc83a5f38728133e3dfa5b0ea2507f076b332b8da29358f27740a86c2f: Status 404 returned error can't find the container with id d0a3acbc83a5f38728133e3dfa5b0ea2507f076b332b8da29358f27740a86c2f Oct 31 00:29:07.662790 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.662769 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9zlgx" Oct 31 00:29:07.697209 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.697187 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:29:07.698290 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.698273 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:29:07.700996 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.700982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m8d27" Oct 31 00:29:07.705944 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:07.705922 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ef0d54_5212_427c_8bab_04b4262c3e11.slice/crio-33e287d9a9d90eea698b1d49ef45eb8ecc3edf7c15489120309e505a674fea5e WatchSource:0}: Error finding container 33e287d9a9d90eea698b1d49ef45eb8ecc3edf7c15489120309e505a674fea5e: Status 404 returned error can't find the container with id 33e287d9a9d90eea698b1d49ef45eb8ecc3edf7c15489120309e505a674fea5e Oct 31 00:29:07.710010 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:07.709985 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbda83c1_6c9e_45d7_9cc5_0ff7f2cc5522.slice/crio-8d965f84be745e3e58f3e0d386b962fca2c6e255395faaef2fc50f2c18a6cd52 WatchSource:0}: Error finding container 8d965f84be745e3e58f3e0d386b962fca2c6e255395faaef2fc50f2c18a6cd52: Status 404 returned error can't find the container with id 8d965f84be745e3e58f3e0d386b962fca2c6e255395faaef2fc50f2c18a6cd52 Oct 31 00:29:07.711427 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:07.711407 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b02c695_144a_4481_bc14_1cd89baba762.slice/crio-0551584673ab68d38c82443ba50bfd39f610979e1c4fa42ddd15024a346d0cd8 WatchSource:0}: Error finding container 0551584673ab68d38c82443ba50bfd39f610979e1c4fa42ddd15024a346d0cd8: Status 404 returned error can't find the container with id 0551584673ab68d38c82443ba50bfd39f610979e1c4fa42ddd15024a346d0cd8 Oct 31 00:29:07.716663 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.716642 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:29:07.725158 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:07.725139 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c29ec2_dcd1_4c23_99cc_66c6684db03a.slice/crio-7a69e6f3ec8dced09bbbf1c906f3b543c04e16205d905423934cd4aff9e18095 WatchSource:0}: Error finding container 7a69e6f3ec8dced09bbbf1c906f3b543c04e16205d905423934cd4aff9e18095: Status 404 returned error can't find the container with id 7a69e6f3ec8dced09bbbf1c906f3b543c04e16205d905423934cd4aff9e18095 Oct 31 00:29:07.739220 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.739196 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:29:07.747913 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:07.747894 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2daade_8f58_42e8_a79b_2a373ff186ea.slice/crio-d6d0ab3c41a935419164bdaf1be1cd9d21b29cd0fda71550a04de8f9c9ca6cc3 WatchSource:0}: Error finding container d6d0ab3c41a935419164bdaf1be1cd9d21b29cd0fda71550a04de8f9c9ca6cc3: Status 404 returned error can't find the container with id d6d0ab3c41a935419164bdaf1be1cd9d21b29cd0fda71550a04de8f9c9ca6cc3 Oct 31 00:29:07.754313 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.754294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:29:07.765017 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.764999 2576 kuberuntime_manager.go:1707] "Unhandled Error" err="container iptables-alerter start failed in pod iptables-alerter-9l4x8_openshift-network-operator(658d1734-2e40-4784-8c85-d57ce4c5c7a8): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 31 00:29:07.765084 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.765031 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-9l4x8" podUID="658d1734-2e40-4784-8c85-d57ce4c5c7a8" Oct 31 00:29:07.930732 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.930645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:07.930732 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:07.930708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:07.930906 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.930754 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:07.930906 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.930796 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:07.930906 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.930829 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs podName:f4b96af0-063c-4752-a290-12371f09d52a nodeName:}" failed. No retries permitted until 2026-10-31 00:29:08.930810644 +0000 UTC m=+2.983782609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs") pod "network-metrics-daemon-87jmt" (UID: "f4b96af0-063c-4752-a290-12371f09d52a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:07.930906 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:07.930849 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret podName:31d4d8b4-ee1c-4703-8ac5-f739d72ab13d nodeName:}" failed. No retries permitted until 2026-10-31 00:29:08.930840585 +0000 UTC m=+2.983812550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret") pod "global-pull-secret-syncer-wj84l" (UID: "31d4d8b4-ee1c-4703-8ac5-f739d72ab13d") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:08.031636 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.031587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkk68\" (UniqueName: \"kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68\") pod \"network-check-target-nsk8c\" (UID: \"698ab510-f0c2-4745-b410-b1697d1aabb8\") " pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:08.031798 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:08.031726 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:29:08.031798 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:08.031747 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:29:08.031798 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:08.031759 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jkk68 for pod openshift-network-diagnostics/network-check-target-nsk8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:08.031918 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:08.031814 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68 podName:698ab510-f0c2-4745-b410-b1697d1aabb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:09.031797876 +0000 UTC m=+3.084769843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkk68" (UniqueName: "kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68") pod "network-check-target-nsk8c" (UID: "698ab510-f0c2-4745-b410-b1697d1aabb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:08.394759 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.394720 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:08.395723 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:08.394863 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:08.425461 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.425393 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m8d27" event={"ID":"0b02c695-144a-4481-bc14-1cd89baba762","Type":"ContainerStarted","Data":"0551584673ab68d38c82443ba50bfd39f610979e1c4fa42ddd15024a346d0cd8"} Oct 31 00:29:08.448233 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.448146 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" event={"ID":"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522","Type":"ContainerStarted","Data":"8d965f84be745e3e58f3e0d386b962fca2c6e255395faaef2fc50f2c18a6cd52"} Oct 31 00:29:08.466021 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.465984 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d4jrp" event={"ID":"089a2aaa-c90a-4df7-8e97-09650f16aab4","Type":"ContainerStarted","Data":"d0a3acbc83a5f38728133e3dfa5b0ea2507f076b332b8da29358f27740a86c2f"} Oct 31 00:29:08.478751 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.478717 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" event={"ID":"912d5cf1-39a7-4fe3-8e72-d6b20bd67476","Type":"ContainerStarted","Data":"583b9b18ba6ebc6dad4e716877a8995f997a662b6214e4cbabdfa9e9744447ca"} Oct 31 00:29:08.483077 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.483044 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9l4x8" event={"ID":"658d1734-2e40-4784-8c85-d57ce4c5c7a8","Type":"ContainerStarted","Data":"70f761047a9fe5bb8263a0f28a78c32a17d695df1f2cac4cc9ffbfa9d240207f"} Oct 31 00:29:08.488537 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:08.488502 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a8b58f028a9a4a0345ad1f809e47fb9913d7f77c731bfd785819e1c71be9472\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-9l4x8" podUID="658d1734-2e40-4784-8c85-d57ce4c5c7a8" Oct 31 00:29:08.508397 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.508338 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqkmw" event={"ID":"50ef0d54-5212-427c-8bab-04b4262c3e11","Type":"ContainerStarted","Data":"33e287d9a9d90eea698b1d49ef45eb8ecc3edf7c15489120309e505a674fea5e"} Oct 31 00:29:08.517435 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.517404 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9zlgx" event={"ID":"db73288c-815d-49ed-9cdb-6799fb6ecc7b","Type":"ContainerStarted","Data":"0fc2d4cf6ec9212d61c8d76159bea8753442415ba2429ea12898622feec41bb9"} Oct 31 00:29:08.538356 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.538255 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" event={"ID":"703d740fe3de84816febee5a3376c36a","Type":"ContainerStarted","Data":"fd51a42ccb70e635a0e592c6baa68e6cb37b6bec9723874747a07385d7d40bf3"} Oct 31 00:29:08.558512 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.558357 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" event={"ID":"8f3794bd171ae09db1b2f1a9b17e1c85","Type":"ContainerStarted","Data":"9d4b6c1858c21c1556fccb4d0402efd73216d33f91e8ddfb133d21de4cf52d0c"} Oct 31 00:29:08.583178 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.582001 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bmn4m" event={"ID":"fb2daade-8f58-42e8-a79b-2a373ff186ea","Type":"ContainerStarted","Data":"d6d0ab3c41a935419164bdaf1be1cd9d21b29cd0fda71550a04de8f9c9ca6cc3"} Oct 31 00:29:08.586677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.586627 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pptjg" event={"ID":"33c29ec2-dcd1-4c23-99cc-66c6684db03a","Type":"ContainerStarted","Data":"7a69e6f3ec8dced09bbbf1c906f3b543c04e16205d905423934cd4aff9e18095"} Oct 31 00:29:08.675734 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.675687 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:29:08.806208 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.806120 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:29:08.943224 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.943186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:08.943424 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:08.943265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:08.943424 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:08.943386 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:08.943550 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:08.943449 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs podName:f4b96af0-063c-4752-a290-12371f09d52a nodeName:}" failed. No retries permitted until 2026-10-31 00:29:10.94342951 +0000 UTC m=+4.996401484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs") pod "network-metrics-daemon-87jmt" (UID: "f4b96af0-063c-4752-a290-12371f09d52a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:08.943926 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:08.943877 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:08.943926 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:08.943930 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret podName:31d4d8b4-ee1c-4703-8ac5-f739d72ab13d nodeName:}" failed. No retries permitted until 2026-10-31 00:29:10.943913675 +0000 UTC m=+4.996885645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret") pod "global-pull-secret-syncer-wj84l" (UID: "31d4d8b4-ee1c-4703-8ac5-f739d72ab13d") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:09.044378 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:09.043856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkk68\" (UniqueName: \"kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68\") pod \"network-check-target-nsk8c\" (UID: \"698ab510-f0c2-4745-b410-b1697d1aabb8\") " pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:09.044378 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:09.043974 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:29:09.044378 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:09.044017 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:29:09.044378 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:09.044027 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jkk68 for pod openshift-network-diagnostics/network-check-target-nsk8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:09.044378 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:09.044086 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68 podName:698ab510-f0c2-4745-b410-b1697d1aabb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:11.044073264 +0000 UTC m=+5.097045231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkk68" (UniqueName: "kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68") pod "network-check-target-nsk8c" (UID: "698ab510-f0c2-4745-b410-b1697d1aabb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:09.392287 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:09.391484 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:09.392287 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:09.391633 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:09.392287 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:09.392058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:09.392287 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:09.392222 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:10.395075 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:10.394989 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:10.395551 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:10.395140 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:10.959025 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:10.958982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:10.959217 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:10.959080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:10.959217 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:10.959175 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:10.959338 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:10.959260 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret podName:31d4d8b4-ee1c-4703-8ac5-f739d72ab13d nodeName:}" failed. No retries permitted until 2026-10-31 00:29:14.959238838 +0000 UTC m=+9.012210808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret") pod "global-pull-secret-syncer-wj84l" (UID: "31d4d8b4-ee1c-4703-8ac5-f739d72ab13d") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:10.959483 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:10.959407 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:10.959554 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:10.959541 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs podName:f4b96af0-063c-4752-a290-12371f09d52a nodeName:}" failed. No retries permitted until 2026-10-31 00:29:14.95952075 +0000 UTC m=+9.012492714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs") pod "network-metrics-daemon-87jmt" (UID: "f4b96af0-063c-4752-a290-12371f09d52a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:11.060495 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:11.059862 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkk68\" (UniqueName: \"kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68\") pod \"network-check-target-nsk8c\" (UID: \"698ab510-f0c2-4745-b410-b1697d1aabb8\") " pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:11.060495 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:11.060029 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:29:11.060495 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:11.060050 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:29:11.060495 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:11.060063 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jkk68 for pod openshift-network-diagnostics/network-check-target-nsk8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:11.060495 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:11.060126 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68 podName:698ab510-f0c2-4745-b410-b1697d1aabb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:15.060105617 +0000 UTC m=+9.113077573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkk68" (UniqueName: "kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68") pod "network-check-target-nsk8c" (UID: "698ab510-f0c2-4745-b410-b1697d1aabb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:11.391548 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:11.391511 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:11.391756 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:11.391685 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:11.392127 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:11.392101 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:11.392237 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:11.392213 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:12.392978 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:12.392760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:12.393400 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:12.393122 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:12.620101 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:12.620054 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" event={"ID":"703d740fe3de84816febee5a3376c36a","Type":"ContainerStarted","Data":"aa51ff09c002cd72f5a0ca4d92fd7b6ef2d5f802af72f3f7fbec94889fa645bc"} Oct 31 00:29:12.627679 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:12.627528 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" event={"ID":"8f3794bd171ae09db1b2f1a9b17e1c85","Type":"ContainerStarted","Data":"dea3b69ae61fd5b8535d5b6417863ac284adb59204d641d3e97e47ca6d45495f"} Oct 31 00:29:12.631523 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:12.631452 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d4jrp" event={"ID":"089a2aaa-c90a-4df7-8e97-09650f16aab4","Type":"ContainerStarted","Data":"3adef8456e01ff17772e5a4efb2e60af1910a19ff9fabe4c9ef6f6256a237995"} Oct 31 00:29:12.636120 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:12.636090 2576 generic.go:356] "Generic (PLEG): container finished" podID="50ef0d54-5212-427c-8bab-04b4262c3e11" containerID="b410a730df7fd813c062d9c68a9399dc24c0923497cf9cb36636f634037cf0a5" exitCode=0 Oct 31 00:29:12.636229 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:12.636160 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqkmw" event={"ID":"50ef0d54-5212-427c-8bab-04b4262c3e11","Type":"ContainerDied","Data":"b410a730df7fd813c062d9c68a9399dc24c0923497cf9cb36636f634037cf0a5"} Oct 31 00:29:12.637147 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:12.637026 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" podStartSLOduration=5.63700843 podStartE2EDuration="5.63700843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:12.636289686 +0000 UTC m=+6.689261670" watchObservedRunningTime="2026-10-31 00:29:12.63700843 +0000 UTC m=+6.689980408" Oct 31 00:29:12.709746 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:12.709692 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5tkpg" Oct 31 00:29:12.718854 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:12.718827 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5tkpg" Oct 31 00:29:13.391922 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:13.391878 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:13.392176 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:13.392030 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:13.392755 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:13.392503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:13.392755 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:13.392619 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:13.641068 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:13.641035 2576 generic.go:356] "Generic (PLEG): container finished" podID="8f3794bd171ae09db1b2f1a9b17e1c85" containerID="dea3b69ae61fd5b8535d5b6417863ac284adb59204d641d3e97e47ca6d45495f" exitCode=0 Oct 31 00:29:13.641491 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:13.641153 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" event={"ID":"8f3794bd171ae09db1b2f1a9b17e1c85","Type":"ContainerDied","Data":"dea3b69ae61fd5b8535d5b6417863ac284adb59204d641d3e97e47ca6d45495f"} Oct 31 00:29:13.664482 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:13.664404 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d4jrp" podStartSLOduration=2.167412711 podStartE2EDuration="6.664383902s" totalImagesPullingTime="4.496971191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:12.691407392 +0000 UTC m=+6.744379369" watchObservedRunningTime="2026-10-31 00:29:13.664383902 +0000 UTC m=+7.717355880" Oct 31 00:29:13.725108 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:13.725044 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:24:12 +0000 UTC" deadline="2028-08-09 01:54:21.393467218 +0000 UTC" Oct 31 00:29:13.725108 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:13.725107 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15553h25m7.668364355s" Oct 31 00:29:14.392989 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:14.392901 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:14.393163 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:14.393061 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:14.648156 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:14.647328 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" event={"ID":"8f3794bd171ae09db1b2f1a9b17e1c85","Type":"ContainerStarted","Data":"52c191c962653ed7e8820d03343a000bea366bb3294568a9ff03869a1eae8a24"} Oct 31 00:29:14.652683 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:14.652190 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bmn4m" event={"ID":"fb2daade-8f58-42e8-a79b-2a373ff186ea","Type":"ContainerStarted","Data":"a4a6673b843e8d521c834b16aea85125fd57511b566311ffb25873fcde92ac47"} Oct 31 00:29:14.654586 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:14.654510 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" event={"ID":"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522","Type":"ContainerStarted","Data":"ba7e982e1c4f94c53724bf76b8f71c93d508d3749ff8138dd38e4a4a9ab198fb"} Oct 31 00:29:14.681864 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:14.681803 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" podStartSLOduration=7.681784459 podStartE2EDuration="7.681784459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:14.665174735 +0000 UTC m=+8.718146712" watchObservedRunningTime="2026-10-31 00:29:14.681784459 +0000 UTC m=+8.734756440" Oct 31 00:29:14.726255 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:14.726180 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:24:12 +0000 UTC" deadline="2028-06-27 20:29:56.671938464 +0000 UTC" Oct 31 00:29:14.726255 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:14.726222 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14540h0m41.945722329s" Oct 31 00:29:14.995231 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:14.995177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:14.995439 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:14.995277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:14.995511 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:14.995465 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:14.995604 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:14.995585 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs podName:f4b96af0-063c-4752-a290-12371f09d52a nodeName:}" failed. No retries permitted until 2026-10-31 00:29:22.995562117 +0000 UTC m=+17.048534092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs") pod "network-metrics-daemon-87jmt" (UID: "f4b96af0-063c-4752-a290-12371f09d52a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:14.996013 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:14.995996 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:14.996094 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:14.996039 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret podName:31d4d8b4-ee1c-4703-8ac5-f739d72ab13d nodeName:}" failed. No retries permitted until 2026-10-31 00:29:22.996027596 +0000 UTC m=+17.048999549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret") pod "global-pull-secret-syncer-wj84l" (UID: "31d4d8b4-ee1c-4703-8ac5-f739d72ab13d") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:15.096124 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:15.096081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkk68\" (UniqueName: \"kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68\") pod \"network-check-target-nsk8c\" (UID: \"698ab510-f0c2-4745-b410-b1697d1aabb8\") " pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:15.096322 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:15.096291 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:29:15.096322 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:15.096318 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:29:15.096461 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:15.096332 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jkk68 for pod openshift-network-diagnostics/network-check-target-nsk8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:15.096461 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:15.096397 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68 podName:698ab510-f0c2-4745-b410-b1697d1aabb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:23.096376827 +0000 UTC m=+17.149348796 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkk68" (UniqueName: "kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68") pod "network-check-target-nsk8c" (UID: "698ab510-f0c2-4745-b410-b1697d1aabb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:15.392143 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:15.392061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:15.392327 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:15.392207 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:15.392682 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:15.392660 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:15.392826 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:15.392801 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:15.732907 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:15.732853 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:29:15.935933 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:15.935885 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:29:15.937576 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:15.937321 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:29:15.956832 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:15.956737 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bmn4m" podStartSLOduration=3.172452126 podStartE2EDuration="8.95671549s" totalImagesPullingTime="5.784263364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:14.681550732 +0000 UTC m=+8.734522709" watchObservedRunningTime="2026-10-31 00:29:15.95671549 +0000 UTC m=+10.009687470" Oct 31 00:29:16.387951 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:16.387864 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:29:15.732905016Z","UUID":"0af616db-67c3-4bfe-b6a0-7e3331a28435","Handler":null,"Name":"","Endpoint":""} Oct 31 00:29:16.389594 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:16.389571 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:29:16.389751 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:16.389604 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:29:16.396861 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:16.396677 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:16.397065 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:16.397036 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:16.663263 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:16.662997 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqkmw" event={"ID":"50ef0d54-5212-427c-8bab-04b4262c3e11","Type":"ContainerStarted","Data":"76063ad4ae8f09d7a6b060bc21392dba7b145f095ae3a3bf85767b99f8afcde0"} Oct 31 00:29:16.665090 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:16.665058 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9zlgx" event={"ID":"db73288c-815d-49ed-9cdb-6799fb6ecc7b","Type":"ContainerStarted","Data":"8f696345ec91f053cca4b9de6b30f33bc3a82139fda114e311d964ad79cb4131"} Oct 31 00:29:16.668257 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:16.668218 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m8d27" event={"ID":"0b02c695-144a-4481-bc14-1cd89baba762","Type":"ContainerStarted","Data":"4c6a41c9f8c29320af61a88950e8ce8e605011fc4cfd179479cdaf43ddc96490"} Oct 31 00:29:16.671332 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:16.671308 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" event={"ID":"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522","Type":"ContainerStarted","Data":"f43b141fe55a26d5fd1b8401919310c98a4557d65ce4f509b314839f9f974b6d"} Oct 31 00:29:16.671412 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:16.671369 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:29:16.671806 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:16.671791 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:29:16.711989 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:16.711943 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-m8d27" podStartSLOduration=1.583674102 podStartE2EDuration="9.711928512s" totalImagesPullingTime="8.12825441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:16.696108136 +0000 UTC m=+10.749080104" watchObservedRunningTime="2026-10-31 00:29:16.711928512 +0000 UTC m=+10.764900484" Oct 31 00:29:16.729247 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:16.729193 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9zlgx" podStartSLOduration=1.610080429 podStartE2EDuration="9.729173248s" totalImagesPullingTime="8.119092819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:16.728384985 +0000 UTC m=+10.781356969" watchObservedRunningTime="2026-10-31 00:29:16.729173248 +0000 UTC m=+10.782145221" Oct 31 00:29:17.391457 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:17.391426 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:17.391915 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:17.391437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:17.391915 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:17.391557 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:17.391915 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:17.391663 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:17.676423 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:17.676385 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" event={"ID":"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522","Type":"ContainerStarted","Data":"c45164e940e37997b124f1dbfad3b3269c797a4890882cfff91a73bce5af0dae"} Oct 31 00:29:17.678492 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:17.678458 2576 generic.go:356] "Generic (PLEG): container finished" podID="50ef0d54-5212-427c-8bab-04b4262c3e11" containerID="76063ad4ae8f09d7a6b060bc21392dba7b145f095ae3a3bf85767b99f8afcde0" exitCode=0 Oct 31 00:29:17.678718 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:17.678689 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqkmw" event={"ID":"50ef0d54-5212-427c-8bab-04b4262c3e11","Type":"ContainerDied","Data":"76063ad4ae8f09d7a6b060bc21392dba7b145f095ae3a3bf85767b99f8afcde0"} Oct 31 00:29:17.695287 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:17.695239 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" podStartSLOduration=1.961855848 podStartE2EDuration="10.695220639s" totalImagesPullingTime="8.733364791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:17.694814049 +0000 UTC m=+11.747786024" watchObservedRunningTime="2026-10-31 00:29:17.695220639 +0000 UTC m=+11.748192618" Oct 31 00:29:18.391972 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:18.391891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:18.392483 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:18.392037 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:18.682416 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:18.682387 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pptjg" event={"ID":"33c29ec2-dcd1-4c23-99cc-66c6684db03a","Type":"ContainerStarted","Data":"a84dcafdcc95b1f3a12f2286173a5928bc6e7d423addb15e35a1527c6875c733"} Oct 31 00:29:18.699262 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:18.699222 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pptjg" podStartSLOduration=1.351319112 podStartE2EDuration="11.699207761s" totalImagesPullingTime="10.347888649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:18.698892472 +0000 UTC m=+12.751864446" watchObservedRunningTime="2026-10-31 00:29:18.699207761 +0000 UTC m=+12.752179735" Oct 31 00:29:19.391710 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:19.391681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:19.391710 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:19.391695 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:19.392177 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:19.392152 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:19.395110 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:19.395084 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:19.687029 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:19.686988 2576 generic.go:356] "Generic (PLEG): container finished" podID="50ef0d54-5212-427c-8bab-04b4262c3e11" containerID="28d59981121f426170fc967c89ee4842380442b1a0360fb6821da00022c4c40c" exitCode=0 Oct 31 00:29:19.687208 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:19.687078 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqkmw" event={"ID":"50ef0d54-5212-427c-8bab-04b4262c3e11","Type":"ContainerDied","Data":"28d59981121f426170fc967c89ee4842380442b1a0360fb6821da00022c4c40c"} Oct 31 00:29:20.391701 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:20.391665 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:20.391873 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:20.391787 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:21.391885 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:21.391852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:21.392364 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:21.392032 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:21.392364 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:21.392075 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:21.392364 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:21.392157 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:21.693411 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:21.693378 2576 generic.go:356] "Generic (PLEG): container finished" podID="50ef0d54-5212-427c-8bab-04b4262c3e11" containerID="4f469a9b112e981dd4d46e9f76e40ec0026e235cb17ae824ce588062eacfcb68" exitCode=0 Oct 31 00:29:21.693577 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:21.693446 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqkmw" event={"ID":"50ef0d54-5212-427c-8bab-04b4262c3e11","Type":"ContainerDied","Data":"4f469a9b112e981dd4d46e9f76e40ec0026e235cb17ae824ce588062eacfcb68"} Oct 31 00:29:22.391917 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:22.391771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:22.391917 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:22.391910 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:23.055039 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.054831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:23.055191 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:23.055045 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:23.055191 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.055091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:23.055191 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:23.055116 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs podName:f4b96af0-063c-4752-a290-12371f09d52a nodeName:}" failed. No retries permitted until 2026-10-31 00:29:39.055094341 +0000 UTC m=+33.108066303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs") pod "network-metrics-daemon-87jmt" (UID: "f4b96af0-063c-4752-a290-12371f09d52a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:23.055191 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:23.055180 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:23.055356 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:23.055220 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret podName:31d4d8b4-ee1c-4703-8ac5-f739d72ab13d nodeName:}" failed. No retries permitted until 2026-10-31 00:29:39.055208368 +0000 UTC m=+33.108180319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret") pod "global-pull-secret-syncer-wj84l" (UID: "31d4d8b4-ee1c-4703-8ac5-f739d72ab13d") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:23.156658 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.156598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkk68\" (UniqueName: \"kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68\") pod \"network-check-target-nsk8c\" (UID: \"698ab510-f0c2-4745-b410-b1697d1aabb8\") " pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:23.156832 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:23.156810 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:29:23.156832 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:23.156833 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:29:23.157016 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:23.156843 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jkk68 for pod openshift-network-diagnostics/network-check-target-nsk8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:23.157016 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:23.156888 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68 podName:698ab510-f0c2-4745-b410-b1697d1aabb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:39.156873887 +0000 UTC m=+33.209845854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkk68" (UniqueName: "kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68") pod "network-check-target-nsk8c" (UID: "698ab510-f0c2-4745-b410-b1697d1aabb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:23.391849 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.391775 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:23.392001 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:23.391884 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:23.392001 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.391926 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:23.392529 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:23.392040 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:23.701046 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.701018 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" event={"ID":"912d5cf1-39a7-4fe3-8e72-d6b20bd67476","Type":"ContainerStarted","Data":"1f154179bab1236dde844bfe9a2463012c784022d9e9b2bf110a1f50acbd9124"} Oct 31 00:29:23.701191 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.701088 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" event={"ID":"912d5cf1-39a7-4fe3-8e72-d6b20bd67476","Type":"ContainerStarted","Data":"8b9e16e932137bf1039a986df3a9ddcaa3d059e3352e4b21bca9257fbafd231d"} Oct 31 00:29:23.701191 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.701133 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" event={"ID":"912d5cf1-39a7-4fe3-8e72-d6b20bd67476","Type":"ContainerStarted","Data":"32ad7152f5084867c9cc6156cf2b3a934abc705f6fe59d1080c39113ea8786b5"} Oct 31 00:29:23.701191 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.701170 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" event={"ID":"912d5cf1-39a7-4fe3-8e72-d6b20bd67476","Type":"ContainerStarted","Data":"5aa3a60be4f89875a3ff66f939b2429ef2ab32adee9772755fbf802a22293735"} Oct 31 00:29:23.701308 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.701198 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" event={"ID":"912d5cf1-39a7-4fe3-8e72-d6b20bd67476","Type":"ContainerStarted","Data":"5c840226faa4a25847a1506c9149210919134c1acae89e9a5222c78e8c5faa74"} Oct 31 00:29:23.701308 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.701226 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" event={"ID":"912d5cf1-39a7-4fe3-8e72-d6b20bd67476","Type":"ContainerStarted","Data":"293d2664ee8b70f331a5735e9b5d02c8742f6756b784a44c512cfa697df10b31"} Oct 31 00:29:23.704940 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.704918 2576 generic.go:356] "Generic (PLEG): container finished" podID="50ef0d54-5212-427c-8bab-04b4262c3e11" containerID="1a59a881970d973991f963d9fa7d98f9c100c25b23ee421f92a4d2302e70eb3f" exitCode=0 Oct 31 00:29:23.705021 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:23.704960 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqkmw" event={"ID":"50ef0d54-5212-427c-8bab-04b4262c3e11","Type":"ContainerDied","Data":"1a59a881970d973991f963d9fa7d98f9c100c25b23ee421f92a4d2302e70eb3f"} Oct 31 00:29:24.392256 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:24.392222 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:24.392652 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:24.392365 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:24.711022 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:24.710992 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9l4x8" event={"ID":"658d1734-2e40-4784-8c85-d57ce4c5c7a8","Type":"ContainerStarted","Data":"8f64925abf48f518a0db0eddb71df7590b0e5029428dda6f667d48d7bf1dcf27"} Oct 31 00:29:24.713920 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:24.713894 2576 generic.go:356] "Generic (PLEG): container finished" podID="50ef0d54-5212-427c-8bab-04b4262c3e11" containerID="f1d51eb145745778d1e747fdeb2a811092f425e822389c9eaee01d3a67032164" exitCode=0 Oct 31 00:29:24.714018 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:24.713936 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqkmw" event={"ID":"50ef0d54-5212-427c-8bab-04b4262c3e11","Type":"ContainerDied","Data":"f1d51eb145745778d1e747fdeb2a811092f425e822389c9eaee01d3a67032164"} Oct 31 00:29:24.728241 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:24.728203 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9l4x8" podStartSLOduration=3.098578013 podStartE2EDuration="17.728193096s" totalImagesPullingTime="14.629615083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:24.728178042 +0000 UTC m=+18.781150015" watchObservedRunningTime="2026-10-31 00:29:24.728193096 +0000 UTC m=+18.781165069" Oct 31 00:29:25.047648 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:25.047556 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m8d27_0b02c695-144a-4481-bc14-1cd89baba762/dns-node-resolver/0.log" Oct 31 00:29:25.392002 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:25.391801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:25.392156 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:25.391853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:25.392156 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:25.392044 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:25.392156 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:25.392145 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:25.719723 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:25.719680 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" event={"ID":"912d5cf1-39a7-4fe3-8e72-d6b20bd67476","Type":"ContainerStarted","Data":"2ac7709d75429386cccef2a7596b0fd4a4960b4bc73201f113caf315d5aadfc1"} Oct 31 00:29:25.722725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:25.722700 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqkmw" event={"ID":"50ef0d54-5212-427c-8bab-04b4262c3e11","Type":"ContainerStarted","Data":"a1168f8d5b22c320361a83caa1b862373ef13abf19c5b07469cb996d615fa532"} Oct 31 00:29:25.742661 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:25.742599 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jqkmw" podStartSLOduration=7.87659856 podStartE2EDuration="18.742588226s" totalImagesPullingTime="10.865989666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:25.742390107 +0000 UTC m=+19.795362080" watchObservedRunningTime="2026-10-31 00:29:25.742588226 +0000 UTC m=+19.795560198" Oct 31 00:29:25.828256 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:25.828235 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmn4m_fb2daade-8f58-42e8-a79b-2a373ff186ea/node-ca/0.log" Oct 31 00:29:26.394076 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:26.394048 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:26.394255 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:26.394138 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:27.392069 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:27.392008 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:27.392826 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:27.392045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:27.392826 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:27.392185 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:27.392826 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:27.392225 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:27.731266 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:27.731081 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" event={"ID":"912d5cf1-39a7-4fe3-8e72-d6b20bd67476","Type":"ContainerStarted","Data":"44d4bdb7d7db34739072526c9b47a38f10c726411f67888bbe3ab156b4f624d2"} Oct 31 00:29:27.731480 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:27.731321 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:27.731480 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:27.731354 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:27.731480 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:27.731395 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:27.747868 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:27.747845 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:27.747971 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:27.747939 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:29:27.755950 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:27.755911 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" podStartSLOduration=5.701451707 podStartE2EDuration="20.755901367s" totalImagesPullingTime="15.05444966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:27.755742144 +0000 UTC m=+21.808714116" watchObservedRunningTime="2026-10-31 00:29:27.755901367 +0000 UTC m=+21.808873353" Oct 31 00:29:28.392010 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:28.391979 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:28.392166 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:28.392082 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:29.392169 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:29.392144 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:29.392610 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:29.392152 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:29.392610 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:29.392231 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:29.392610 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:29.392329 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:30.391690 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:30.391662 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:30.391869 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:30.391757 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:31.391751 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:31.391720 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:31.392239 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:31.391887 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:31.392239 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:31.391921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:31.392239 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:31.391987 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:32.392243 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:32.392214 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:32.392668 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:32.392342 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:33.391427 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:33.391396 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:33.391582 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:33.391398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:33.391582 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:33.391499 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:33.391668 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:33.391586 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:34.396597 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:34.396570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:34.396993 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:34.396676 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:35.391675 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:35.391650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:35.391825 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:35.391753 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:35.391825 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:35.391813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:35.391916 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:35.391899 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:36.392396 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:36.392371 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:36.392874 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:36.392456 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:37.391644 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:37.391596 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:37.391836 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:37.391732 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:37.391836 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:37.391781 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:37.391948 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:37.391859 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:38.392300 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:38.392261 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:38.392828 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:38.392395 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:39.071033 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:39.071002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:39.071192 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:39.071050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:39.071192 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:39.071181 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:39.071269 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:39.071188 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:39.071269 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:39.071240 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs podName:f4b96af0-063c-4752-a290-12371f09d52a nodeName:}" failed. No retries permitted until 2026-10-31 00:30:11.071225345 +0000 UTC m=+65.124197296 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs") pod "network-metrics-daemon-87jmt" (UID: "f4b96af0-063c-4752-a290-12371f09d52a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:39.071269 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:39.071255 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret podName:31d4d8b4-ee1c-4703-8ac5-f739d72ab13d nodeName:}" failed. No retries permitted until 2026-10-31 00:30:11.071247988 +0000 UTC m=+65.124219938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret") pod "global-pull-secret-syncer-wj84l" (UID: "31d4d8b4-ee1c-4703-8ac5-f739d72ab13d") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:39.172001 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:39.171972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkk68\" (UniqueName: \"kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68\") pod \"network-check-target-nsk8c\" (UID: \"698ab510-f0c2-4745-b410-b1697d1aabb8\") " pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:39.172159 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:39.172088 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:29:39.172159 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:39.172104 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:29:39.172159 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:39.172113 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jkk68 for pod openshift-network-diagnostics/network-check-target-nsk8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:39.172256 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:39.172163 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68 podName:698ab510-f0c2-4745-b410-b1697d1aabb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:11.1721492 +0000 UTC m=+65.225121152 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkk68" (UniqueName: "kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68") pod "network-check-target-nsk8c" (UID: "698ab510-f0c2-4745-b410-b1697d1aabb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:39.392051 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:39.391968 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:39.392194 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:39.391968 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:39.392194 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:39.392093 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:39.392194 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:39.392152 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:40.392183 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:40.392154 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:40.392548 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:40.392258 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:41.391950 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:41.391918 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:41.392111 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:41.392034 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:41.392111 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:41.392038 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:41.392197 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:41.392128 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:42.033858 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:42.033827 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wj84l"] Oct 31 00:29:42.034335 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:42.033985 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:42.034335 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:42.034075 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:42.041826 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:42.041746 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nsk8c"] Oct 31 00:29:42.041826 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:42.041812 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87jmt"] Oct 31 00:29:42.042007 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:42.041909 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:42.042064 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:42.042015 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:42.042238 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:42.042216 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:42.042327 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:42.042308 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:43.340387 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:43.340314 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m8d27_0b02c695-144a-4481-bc14-1cd89baba762/dns-node-resolver/0.log" Oct 31 00:29:43.391469 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:43.391438 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:43.391469 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:43.391467 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:43.391689 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:43.391472 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:43.391689 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:43.391539 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:43.391689 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:43.391610 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:43.391821 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:43.391728 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:44.340384 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:44.340357 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmn4m_fb2daade-8f58-42e8-a79b-2a373ff186ea/node-ca/0.log" Oct 31 00:29:45.392014 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:45.391824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:45.392417 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:45.391824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:45.392417 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:45.392096 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wj84l" podUID="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" Oct 31 00:29:45.392417 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:45.391848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:45.392417 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:45.392183 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsk8c" podUID="698ab510-f0c2-4745-b410-b1697d1aabb8" Oct 31 00:29:45.392417 ip-10-0-134-252 kubenswrapper[2576]: E1031 00:29:45.392256 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jmt" podUID="f4b96af0-063c-4752-a290-12371f09d52a" Oct 31 00:29:46.720005 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.719975 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeReady" Oct 31 00:29:46.720392 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.720100 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:29:46.769935 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.769909 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g62jt"] Oct 31 00:29:46.787852 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.787816 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8ssc4"] Oct 31 00:29:46.788064 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.787952 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:46.792006 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.791977 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:29:46.792173 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.792014 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:29:46.792173 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.792085 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pcfpx\"" Oct 31 00:29:46.792173 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.792117 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:29:46.792396 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.792281 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:29:46.805796 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.805776 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8ssc4"] Oct 31 00:29:46.805887 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.805855 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8ssc4" Oct 31 00:29:46.805887 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.805862 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g62jt"] Oct 31 00:29:46.805960 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.805910 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xbdxj"] Oct 31 00:29:46.819793 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.819773 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xbdxj"] Oct 31 00:29:46.819880 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.819862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:46.892599 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.892569 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:29:46.892826 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.892792 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:29:46.892950 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.892934 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zxj7m\"" Oct 31 00:29:46.893008 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.892964 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:29:46.893008 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.892976 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:29:46.893086 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.893062 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:29:46.893119 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.893111 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-trzmz\"" Oct 31 00:29:46.930448 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.930416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97194d65-9d7e-47b2-8c55-2283301e8a4d-config-volume\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:46.930540 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.930451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f79572d1-5d21-4771-b613-85a421e550b3-data-volume\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:46.930540 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.930473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97194d65-9d7e-47b2-8c55-2283301e8a4d-metrics-tls\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:46.930540 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.930504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f79572d1-5d21-4771-b613-85a421e550b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:46.930646 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.930562 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64kwx\" (UniqueName: \"kubernetes.io/projected/f79572d1-5d21-4771-b613-85a421e550b3-kube-api-access-64kwx\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:46.930646 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.930585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97194d65-9d7e-47b2-8c55-2283301e8a4d-tmp-dir\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:46.930738 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.930717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f79572d1-5d21-4771-b613-85a421e550b3-crio-socket\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:46.930789 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.930775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjn5\" (UniqueName: \"kubernetes.io/projected/226527e6-3023-4c5a-b600-fb6322b77877-kube-api-access-dwjn5\") pod \"ingress-canary-8ssc4\" (UID: \"226527e6-3023-4c5a-b600-fb6322b77877\") " pod="openshift-ingress-canary/ingress-canary-8ssc4" Oct 31 00:29:46.930855 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.930842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/226527e6-3023-4c5a-b600-fb6322b77877-cert\") pod \"ingress-canary-8ssc4\" (UID: \"226527e6-3023-4c5a-b600-fb6322b77877\") " pod="openshift-ingress-canary/ingress-canary-8ssc4" Oct 31 00:29:46.930885 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.930863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfvq9\" (UniqueName: \"kubernetes.io/projected/97194d65-9d7e-47b2-8c55-2283301e8a4d-kube-api-access-qfvq9\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:46.930885 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:46.930880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f79572d1-5d21-4771-b613-85a421e550b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:47.031821 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.031740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f79572d1-5d21-4771-b613-85a421e550b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:47.031821 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.031787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64kwx\" (UniqueName: \"kubernetes.io/projected/f79572d1-5d21-4771-b613-85a421e550b3-kube-api-access-64kwx\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:47.031821 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.031807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97194d65-9d7e-47b2-8c55-2283301e8a4d-tmp-dir\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:47.032022 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.031829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f79572d1-5d21-4771-b613-85a421e550b3-crio-socket\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:47.032022 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.031870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjn5\" (UniqueName: \"kubernetes.io/projected/226527e6-3023-4c5a-b600-fb6322b77877-kube-api-access-dwjn5\") pod \"ingress-canary-8ssc4\" (UID: \"226527e6-3023-4c5a-b600-fb6322b77877\") " pod="openshift-ingress-canary/ingress-canary-8ssc4" Oct 31 00:29:47.032022 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.031896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/226527e6-3023-4c5a-b600-fb6322b77877-cert\") pod \"ingress-canary-8ssc4\" (UID: \"226527e6-3023-4c5a-b600-fb6322b77877\") " pod="openshift-ingress-canary/ingress-canary-8ssc4" Oct 31 00:29:47.032022 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.031914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfvq9\" (UniqueName: \"kubernetes.io/projected/97194d65-9d7e-47b2-8c55-2283301e8a4d-kube-api-access-qfvq9\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:47.032022 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.031940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f79572d1-5d21-4771-b613-85a421e550b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:47.032022 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.031970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97194d65-9d7e-47b2-8c55-2283301e8a4d-config-volume\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:47.032022 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.031995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f79572d1-5d21-4771-b613-85a421e550b3-data-volume\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:47.032022 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.032018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97194d65-9d7e-47b2-8c55-2283301e8a4d-metrics-tls\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:47.032371 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.032214 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f79572d1-5d21-4771-b613-85a421e550b3-crio-socket\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:47.032469 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.032442 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f79572d1-5d21-4771-b613-85a421e550b3-data-volume\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:47.032550 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.032505 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97194d65-9d7e-47b2-8c55-2283301e8a4d-tmp-dir\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:47.032825 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.032805 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97194d65-9d7e-47b2-8c55-2283301e8a4d-config-volume\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:47.032925 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.032849 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f79572d1-5d21-4771-b613-85a421e550b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:47.036084 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.036067 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97194d65-9d7e-47b2-8c55-2283301e8a4d-metrics-tls\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:47.036158 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.036067 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f79572d1-5d21-4771-b613-85a421e550b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:47.036267 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.036240 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/226527e6-3023-4c5a-b600-fb6322b77877-cert\") pod \"ingress-canary-8ssc4\" (UID: \"226527e6-3023-4c5a-b600-fb6322b77877\") " pod="openshift-ingress-canary/ingress-canary-8ssc4" Oct 31 00:29:47.041367 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.041344 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64kwx\" (UniqueName: \"kubernetes.io/projected/f79572d1-5d21-4771-b613-85a421e550b3-kube-api-access-64kwx\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:47.041593 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.041573 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfvq9\" (UniqueName: \"kubernetes.io/projected/97194d65-9d7e-47b2-8c55-2283301e8a4d-kube-api-access-qfvq9\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:47.042531 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.042514 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjn5\" (UniqueName: \"kubernetes.io/projected/226527e6-3023-4c5a-b600-fb6322b77877-kube-api-access-dwjn5\") pod \"ingress-canary-8ssc4\" (UID: \"226527e6-3023-4c5a-b600-fb6322b77877\") " pod="openshift-ingress-canary/ingress-canary-8ssc4" Oct 31 00:29:47.140712 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.140678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:29:47.169097 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.169068 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8ssc4" Oct 31 00:29:47.191141 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.191119 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:47.391643 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.391533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:29:47.391643 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.391577 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:29:47.391858 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.391682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:29:47.397030 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.397005 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:29:47.397163 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.397075 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xkpl6\"" Oct 31 00:29:47.397163 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.397094 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:29:47.397163 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.397143 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:29:47.397876 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.397854 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-brqzn\"" Oct 31 00:29:47.397959 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.397857 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9g68m\"" Oct 31 00:29:47.397959 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.397880 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:29:47.497949 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.497909 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8ssc4"] Oct 31 00:29:47.499370 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.499344 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xbdxj"] Oct 31 00:29:47.499903 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.499884 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g62jt"] Oct 31 00:29:47.503378 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:47.503351 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226527e6_3023_4c5a_b600_fb6322b77877.slice/crio-623f37df70210507f4c3f43133c9802b978f52ccc4269ddb66a60d04898f6dbd WatchSource:0}: Error finding container 623f37df70210507f4c3f43133c9802b978f52ccc4269ddb66a60d04898f6dbd: Status 404 returned error can't find the container with id 623f37df70210507f4c3f43133c9802b978f52ccc4269ddb66a60d04898f6dbd Oct 31 00:29:47.503705 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:47.503684 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97194d65_9d7e_47b2_8c55_2283301e8a4d.slice/crio-b765ff17367f70849cc916487c1410916d39fd9c8054300bcf050cec9c75dcf2 WatchSource:0}: Error finding container b765ff17367f70849cc916487c1410916d39fd9c8054300bcf050cec9c75dcf2: Status 404 returned error can't find the container with id b765ff17367f70849cc916487c1410916d39fd9c8054300bcf050cec9c75dcf2 Oct 31 00:29:47.504731 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:47.504708 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79572d1_5d21_4771_b613_85a421e550b3.slice/crio-66ae9e7e3ff66cbb21d9c557544e24d5c884109124fc1560576160869237dc2c WatchSource:0}: Error finding container 66ae9e7e3ff66cbb21d9c557544e24d5c884109124fc1560576160869237dc2c: Status 404 returned error can't find the container with id 66ae9e7e3ff66cbb21d9c557544e24d5c884109124fc1560576160869237dc2c Oct 31 00:29:47.771562 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.771359 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbdxj" event={"ID":"97194d65-9d7e-47b2-8c55-2283301e8a4d","Type":"ContainerStarted","Data":"b765ff17367f70849cc916487c1410916d39fd9c8054300bcf050cec9c75dcf2"} Oct 31 00:29:47.772465 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.772434 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8ssc4" event={"ID":"226527e6-3023-4c5a-b600-fb6322b77877","Type":"ContainerStarted","Data":"623f37df70210507f4c3f43133c9802b978f52ccc4269ddb66a60d04898f6dbd"} Oct 31 00:29:47.773728 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.773709 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g62jt" event={"ID":"f79572d1-5d21-4771-b613-85a421e550b3","Type":"ContainerStarted","Data":"ca91c64eb14ad034676ed5b021abbbfe0b2031c6680f76f793d693b0e6b3f0bd"} Oct 31 00:29:47.773815 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:47.773746 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g62jt" event={"ID":"f79572d1-5d21-4771-b613-85a421e550b3","Type":"ContainerStarted","Data":"66ae9e7e3ff66cbb21d9c557544e24d5c884109124fc1560576160869237dc2c"} Oct 31 00:29:49.783488 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:49.783397 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g62jt" event={"ID":"f79572d1-5d21-4771-b613-85a421e550b3","Type":"ContainerStarted","Data":"d027b28700fc5a9e028cf40b0a4e6f293dc1380d3996461085f065e30c7163c9"} Oct 31 00:29:49.786583 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:49.785894 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbdxj" event={"ID":"97194d65-9d7e-47b2-8c55-2283301e8a4d","Type":"ContainerStarted","Data":"8ad45cacb7d28984dfb9f73548e549e8a3cfcf0353ad59039e348cc6714fc5ff"} Oct 31 00:29:49.786583 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:49.785937 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbdxj" event={"ID":"97194d65-9d7e-47b2-8c55-2283301e8a4d","Type":"ContainerStarted","Data":"b6ff26c6e4abbc96080e9785f933a3374826e509413846daccbc8cee321d362b"} Oct 31 00:29:49.786583 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:49.786499 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xbdxj" Oct 31 00:29:49.806022 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:49.805970 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-xbdxj" podStartSLOduration=1.8163769520000002 podStartE2EDuration="3.805955776s" totalImagesPullingTime="1.989578824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:49.8047016 +0000 UTC m=+43.857673574" watchObservedRunningTime="2026-10-31 00:29:49.805955776 +0000 UTC m=+43.858927745" Oct 31 00:29:49.977680 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:49.977651 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb"] Oct 31 00:29:49.994013 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:49.992891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:49.996074 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:49.996048 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb"] Oct 31 00:29:50.001702 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.001678 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4"] Oct 31 00:29:50.002896 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.002875 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:29:50.003041 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.003026 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:29:50.003115 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.003099 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:29:50.003833 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.003813 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kvd4r\"" Oct 31 00:29:50.003949 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.003935 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:29:50.004106 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.004084 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:29:50.015288 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.015240 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.017609 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.017588 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7jm8z"] Oct 31 00:29:50.031677 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.031651 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4"] Oct 31 00:29:50.031787 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.031738 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.101495 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.101467 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gm9sb\"" Oct 31 00:29:50.101654 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.101537 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:29:50.101654 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.101609 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:29:50.101774 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.101730 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z8njm\"" Oct 31 00:29:50.101837 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.101791 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:29:50.101837 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.101817 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:29:50.101948 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.101899 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:29:50.101948 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.101928 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:29:50.156872 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.156843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-sys\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.157012 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.156883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.157012 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.156906 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-textfile\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.157012 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.156923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v24jd\" (UniqueName: \"kubernetes.io/projected/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-api-access-v24jd\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.157012 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.156949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.157012 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.156975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fba123f2-dd49-407a-8e1d-c568c0d66ce4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.157012 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.156998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.157224 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.157019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.157224 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.157060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmrdz\" (UniqueName: \"kubernetes.io/projected/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-kube-api-access-zmrdz\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.157224 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.157086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-tls\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.157224 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.157109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.157224 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.157123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-root\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.157224 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.157139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b22fa0b3-c20e-402b-81b5-898a838d42c9-metrics-client-ca\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.157224 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.157166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.157224 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.157188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.157224 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.157202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fba123f2-dd49-407a-8e1d-c568c0d66ce4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.157224 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.157215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq7sf\" (UniqueName: \"kubernetes.io/projected/b22fa0b3-c20e-402b-81b5-898a838d42c9-kube-api-access-pq7sf\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.157526 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.157232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.157526 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.157256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-wtmp\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.257989 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.257943 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.257989 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.257993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-root\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.258221 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b22fa0b3-c20e-402b-81b5-898a838d42c9-metrics-client-ca\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.258221 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.258221 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.258221 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fba123f2-dd49-407a-8e1d-c568c0d66ce4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.258221 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7sf\" (UniqueName: \"kubernetes.io/projected/b22fa0b3-c20e-402b-81b5-898a838d42c9-kube-api-access-pq7sf\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.258221 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.258221 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-wtmp\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.258547 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-sys\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.258547 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.258547 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-textfile\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.258547 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v24jd\" (UniqueName: \"kubernetes.io/projected/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-api-access-v24jd\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.258547 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.258547 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fba123f2-dd49-407a-8e1d-c568c0d66ce4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.258547 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.258547 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.258547 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmrdz\" (UniqueName: \"kubernetes.io/projected/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-kube-api-access-zmrdz\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.258547 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-tls\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.259040 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258724 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-root\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.259040 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258732 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.259040 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258783 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.259040 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258906 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b22fa0b3-c20e-402b-81b5-898a838d42c9-metrics-client-ca\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.259040 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.258924 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-wtmp\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.259307 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.259257 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fba123f2-dd49-407a-8e1d-c568c0d66ce4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.259498 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.259476 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-textfile\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.259551 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.259514 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fba123f2-dd49-407a-8e1d-c568c0d66ce4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.259598 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.259561 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-sys\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.260441 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.260415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.262990 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.262911 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.263105 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.263045 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.263105 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.263055 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.263213 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.263140 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.263605 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.263565 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.264877 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.264851 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-tls\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.268176 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.268114 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmrdz\" (UniqueName: \"kubernetes.io/projected/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-kube-api-access-zmrdz\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.268176 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.268156 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7sf\" (UniqueName: \"kubernetes.io/projected/b22fa0b3-c20e-402b-81b5-898a838d42c9-kube-api-access-pq7sf\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.268499 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.268482 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v24jd\" (UniqueName: \"kubernetes.io/projected/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-api-access-v24jd\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.344630 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.344577 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:29:50.387540 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.387508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:29:50.396321 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.396143 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:29:50.407729 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:50.407676 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22fa0b3_c20e_402b_81b5_898a838d42c9.slice/crio-19c09337655a903b6c9d5a935ef64772f2f4f3241dc488fd30039fc4b2b91e6d WatchSource:0}: Error finding container 19c09337655a903b6c9d5a935ef64772f2f4f3241dc488fd30039fc4b2b91e6d: Status 404 returned error can't find the container with id 19c09337655a903b6c9d5a935ef64772f2f4f3241dc488fd30039fc4b2b91e6d Oct 31 00:29:50.663600 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.663459 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb"] Oct 31 00:29:50.667379 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:50.667352 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2acbfb_3c5b_4d40_8365_066e0d29f72e.slice/crio-d7c94a13db71ce4c24ca57fa25974d9ac751d012664ea32f2f27b146deeeeca6 WatchSource:0}: Error finding container d7c94a13db71ce4c24ca57fa25974d9ac751d012664ea32f2f27b146deeeeca6: Status 404 returned error can't find the container with id d7c94a13db71ce4c24ca57fa25974d9ac751d012664ea32f2f27b146deeeeca6 Oct 31 00:29:50.704474 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.704441 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4"] Oct 31 00:29:50.708284 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:50.708256 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba123f2_dd49_407a_8e1d_c568c0d66ce4.slice/crio-9f9d03935951e80f18e6c1b2e8f8463fbcb619f90c4624966a71dd7182c89180 WatchSource:0}: Error finding container 9f9d03935951e80f18e6c1b2e8f8463fbcb619f90c4624966a71dd7182c89180: Status 404 returned error can't find the container with id 9f9d03935951e80f18e6c1b2e8f8463fbcb619f90c4624966a71dd7182c89180 Oct 31 00:29:50.789799 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.789768 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" event={"ID":"fba123f2-dd49-407a-8e1d-c568c0d66ce4","Type":"ContainerStarted","Data":"9f9d03935951e80f18e6c1b2e8f8463fbcb619f90c4624966a71dd7182c89180"} Oct 31 00:29:50.790953 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.790930 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7jm8z" event={"ID":"b22fa0b3-c20e-402b-81b5-898a838d42c9","Type":"ContainerStarted","Data":"19c09337655a903b6c9d5a935ef64772f2f4f3241dc488fd30039fc4b2b91e6d"} Oct 31 00:29:50.792388 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.792360 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" event={"ID":"ea2acbfb-3c5b-4d40-8365-066e0d29f72e","Type":"ContainerStarted","Data":"2b02a3f8c2ad939b2d4831acc1f162f64ebbe974052dc572f327faed7105423f"} Oct 31 00:29:50.792540 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:50.792420 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" event={"ID":"ea2acbfb-3c5b-4d40-8365-066e0d29f72e","Type":"ContainerStarted","Data":"d7c94a13db71ce4c24ca57fa25974d9ac751d012664ea32f2f27b146deeeeca6"} Oct 31 00:29:51.799549 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:51.799357 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g62jt" event={"ID":"f79572d1-5d21-4771-b613-85a421e550b3","Type":"ContainerStarted","Data":"7ada77647561664bf2f1592fd0f3957acbb564273e3d8037a6b33794622432e1"} Oct 31 00:29:51.802228 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:51.802203 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8ssc4" event={"ID":"226527e6-3023-4c5a-b600-fb6322b77877","Type":"ContainerStarted","Data":"9d2ad9e080d22926b2e1c2fe336d3eb4cf82a77912719b11122ada13ed441dc9"} Oct 31 00:29:51.806935 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:51.806908 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" event={"ID":"ea2acbfb-3c5b-4d40-8365-066e0d29f72e","Type":"ContainerStarted","Data":"fbf77bf4f676e6ad40ae675412eafa59a1c1320f3d8a246afaf1393ece8d593b"} Oct 31 00:29:51.840643 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:51.839878 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-g62jt" podStartSLOduration=2.232712374 podStartE2EDuration="5.839859474s" totalImagesPullingTime="3.6071471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:51.820438923 +0000 UTC m=+45.873410896" watchObservedRunningTime="2026-10-31 00:29:51.839859474 +0000 UTC m=+45.892831448" Oct 31 00:29:51.840643 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:51.840029 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8ssc4" podStartSLOduration=2.350771852 podStartE2EDuration="5.840023202s" totalImagesPullingTime="3.48925135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:51.839834918 +0000 UTC m=+45.892806920" watchObservedRunningTime="2026-10-31 00:29:51.840023202 +0000 UTC m=+45.892995177" Oct 31 00:29:52.812485 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:52.812444 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" event={"ID":"ea2acbfb-3c5b-4d40-8365-066e0d29f72e","Type":"ContainerStarted","Data":"ba20487b6e89e3984272e14259f51b8dc2ec2e6dbdd88f39da47940e159961ec"} Oct 31 00:29:52.814740 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:52.814712 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" event={"ID":"fba123f2-dd49-407a-8e1d-c568c0d66ce4","Type":"ContainerStarted","Data":"ed9984bd051f0d02976510229444c706f5e02e8d8fd73d5e037c8d934124e9e6"} Oct 31 00:29:52.814879 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:52.814768 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" event={"ID":"fba123f2-dd49-407a-8e1d-c568c0d66ce4","Type":"ContainerStarted","Data":"5eaff87e140c5fb41456a619fda4df96be2e08670314ec044b650956a67a17bf"} Oct 31 00:29:52.814879 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:52.814796 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" event={"ID":"fba123f2-dd49-407a-8e1d-c568c0d66ce4","Type":"ContainerStarted","Data":"0b09e355349d7a232c05ab201f00463323e5d01bc3afe7a79885f9d4a9b1923b"} Oct 31 00:29:52.816364 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:52.816336 2576 generic.go:356] "Generic (PLEG): container finished" podID="b22fa0b3-c20e-402b-81b5-898a838d42c9" containerID="bc1ab3ca6b68767a8c6a31b043cd01174f78b66193fffbb522774d30014450ff" exitCode=0 Oct 31 00:29:52.816459 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:52.816361 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7jm8z" event={"ID":"b22fa0b3-c20e-402b-81b5-898a838d42c9","Type":"ContainerDied","Data":"bc1ab3ca6b68767a8c6a31b043cd01174f78b66193fffbb522774d30014450ff"} Oct 31 00:29:52.829541 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:52.829493 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" podStartSLOduration=2.361859593 podStartE2EDuration="3.829477737s" totalImagesPullingTime="1.467618144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:52.828505664 +0000 UTC m=+46.881477638" watchObservedRunningTime="2026-10-31 00:29:52.829477737 +0000 UTC m=+46.882449711" Oct 31 00:29:52.846150 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:52.846103 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" podStartSLOduration=2.016083428 podStartE2EDuration="3.846090854s" totalImagesPullingTime="1.830007426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:52.845452598 +0000 UTC m=+46.898424572" watchObservedRunningTime="2026-10-31 00:29:52.846090854 +0000 UTC m=+46.899062826" Oct 31 00:29:53.821897 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:53.821844 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7jm8z" event={"ID":"b22fa0b3-c20e-402b-81b5-898a838d42c9","Type":"ContainerStarted","Data":"25c6faeaf40d3283d2c1e8936980b09e515497fb50f4c8b87a6404aab9d29821"} Oct 31 00:29:53.822299 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:53.821920 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7jm8z" event={"ID":"b22fa0b3-c20e-402b-81b5-898a838d42c9","Type":"ContainerStarted","Data":"755f1aac08afd91c59aa61fbadbf2dd582cd4400442bf051ad2c46e2a9648560"} Oct 31 00:29:53.846384 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:53.846331 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7jm8z" podStartSLOduration=2.687299334 podStartE2EDuration="4.846317441s" totalImagesPullingTime="2.073646464s" totalInitContainerRuntime="85.371643ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:53.844809967 +0000 UTC m=+47.897782295" watchObservedRunningTime="2026-10-31 00:29:53.846317441 +0000 UTC m=+47.899289413" Oct 31 00:29:54.783666 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:54.783633 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm"] Oct 31 00:29:54.786758 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:54.786741 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:29:54.796404 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:54.796385 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:29:54.796542 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:54.796413 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-c87xz\"" Oct 31 00:29:54.796797 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:54.796779 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm"] Oct 31 00:29:54.897334 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:54.897306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjz8r\" (UniqueName: \"kubernetes.io/projected/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e-kube-api-access-gjz8r\") pod \"monitoring-plugin-77f4f76594-rz5vm\" (UID: \"9cd77f3f-3824-44dd-8d5f-156ff99d5b7e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:29:54.897736 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:54.897381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rz5vm\" (UID: \"9cd77f3f-3824-44dd-8d5f-156ff99d5b7e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:29:54.998471 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:54.998431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rz5vm\" (UID: \"9cd77f3f-3824-44dd-8d5f-156ff99d5b7e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:29:54.998691 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:54.998502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjz8r\" (UniqueName: \"kubernetes.io/projected/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e-kube-api-access-gjz8r\") pod \"monitoring-plugin-77f4f76594-rz5vm\" (UID: \"9cd77f3f-3824-44dd-8d5f-156ff99d5b7e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:29:55.002757 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:55.002737 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rz5vm\" (UID: \"9cd77f3f-3824-44dd-8d5f-156ff99d5b7e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:29:55.007233 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:55.007199 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjz8r\" (UniqueName: \"kubernetes.io/projected/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e-kube-api-access-gjz8r\") pod \"monitoring-plugin-77f4f76594-rz5vm\" (UID: \"9cd77f3f-3824-44dd-8d5f-156ff99d5b7e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:29:55.098195 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:55.098130 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:29:55.380220 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:55.380199 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm"] Oct 31 00:29:55.382487 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:55.382460 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd77f3f_3824_44dd_8d5f_156ff99d5b7e.slice/crio-8240f10e9131e2cc38e21c7d60715c22b3de5d759ac772ba3a459b5970804f42 WatchSource:0}: Error finding container 8240f10e9131e2cc38e21c7d60715c22b3de5d759ac772ba3a459b5970804f42: Status 404 returned error can't find the container with id 8240f10e9131e2cc38e21c7d60715c22b3de5d759ac772ba3a459b5970804f42 Oct 31 00:29:55.828223 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:55.828180 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" event={"ID":"9cd77f3f-3824-44dd-8d5f-156ff99d5b7e","Type":"ContainerStarted","Data":"8240f10e9131e2cc38e21c7d60715c22b3de5d759ac772ba3a459b5970804f42"} Oct 31 00:29:56.276479 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.276447 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:29:56.280042 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.280021 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.285086 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285062 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:29:56.285086 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285063 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:29:56.285293 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285091 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:29:56.285293 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285068 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:29:56.285293 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285104 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:29:56.285293 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285237 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fmft110ekd91k\"" Oct 31 00:29:56.285293 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285270 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:29:56.285293 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285286 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:29:56.285497 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285333 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:29:56.285497 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285336 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:29:56.285497 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285359 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:29:56.285497 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285367 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:29:56.285497 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285383 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:29:56.285497 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285415 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jgktj\"" Oct 31 00:29:56.285497 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.285427 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:29:56.290091 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.290072 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:29:56.291466 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.291445 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:29:56.295591 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.295570 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:29:56.409528 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409718 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409718 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409718 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409718 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409718 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409718 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409946 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409946 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409946 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409946 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppsr\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-kube-api-access-vppsr\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409946 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409946 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.409946 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.410174 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.410174 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.409998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.410174 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.410025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.410174 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.410049 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.410174 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.410089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.410174 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.410113 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.511482 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.511482 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.511753 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.511753 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.511753 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.511753 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.511753 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.511753 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.511753 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.511753 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.511753 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppsr\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-kube-api-access-vppsr\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.512174 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.512174 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.511828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.512323 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.512301 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.515903 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.515872 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.516047 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.515943 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.516327 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.516304 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.516739 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.516716 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.516859 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.516733 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.516859 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.516814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.516859 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.516849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.517021 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.516884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.517021 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.516921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.517021 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.516945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.517021 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.516987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.517021 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.517012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.517237 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.517081 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.517237 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.517146 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.517596 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.517572 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.517777 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.517752 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.518079 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.518055 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.518472 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.518450 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.518905 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.518879 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.519193 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.519166 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.519278 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.519191 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.520035 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.519696 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.520134 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.520050 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.520554 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.520536 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.520664 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.520549 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.522302 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.522280 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppsr\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-kube-api-access-vppsr\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.611468 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.611378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:56.833106 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.833069 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" event={"ID":"9cd77f3f-3824-44dd-8d5f-156ff99d5b7e","Type":"ContainerStarted","Data":"ac4624d16cada8e357facd711ceba24b47de56fcf8483bc888ea83060b545a0f"} Oct 31 00:29:56.833278 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.833261 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:29:56.838368 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.838344 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:29:56.851798 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.851760 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" podStartSLOduration=1.494550761 podStartE2EDuration="2.851748922s" totalImagesPullingTime="1.357198161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:56.851039809 +0000 UTC m=+50.904011795" watchObservedRunningTime="2026-10-31 00:29:56.851748922 +0000 UTC m=+50.904720894" Oct 31 00:29:56.908024 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:56.907999 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:29:56.909777 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:29:56.909753 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2e4815_e7d6_44e0_bf9f_62ed9eaee9d2.slice/crio-fcbd30c9a52d9925e349b62c93ae7b8f7e6377aeca2b79dc7f6bea32cc9a8918 WatchSource:0}: Error finding container fcbd30c9a52d9925e349b62c93ae7b8f7e6377aeca2b79dc7f6bea32cc9a8918: Status 404 returned error can't find the container with id fcbd30c9a52d9925e349b62c93ae7b8f7e6377aeca2b79dc7f6bea32cc9a8918 Oct 31 00:29:57.841480 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:57.841445 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerStarted","Data":"fcbd30c9a52d9925e349b62c93ae7b8f7e6377aeca2b79dc7f6bea32cc9a8918"} Oct 31 00:29:58.845822 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:58.845789 2576 generic.go:356] "Generic (PLEG): container finished" podID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerID="3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263" exitCode=0 Oct 31 00:29:58.846202 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:58.845886 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerDied","Data":"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263"} Oct 31 00:29:59.749313 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:29:59.749281 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:00.808763 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:00.808734 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xbdxj" Oct 31 00:30:02.862495 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:02.862462 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerStarted","Data":"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8"} Oct 31 00:30:02.862903 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:02.862528 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerStarted","Data":"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379"} Oct 31 00:30:04.873184 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:04.873145 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerStarted","Data":"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4"} Oct 31 00:30:04.873522 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:04.873227 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerStarted","Data":"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5"} Oct 31 00:30:04.873522 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:04.873266 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerStarted","Data":"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2"} Oct 31 00:30:04.873522 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:04.873304 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerStarted","Data":"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3"} Oct 31 00:30:04.901480 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:04.901412 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6090662780000002 podStartE2EDuration="8.901394314s" totalImagesPullingTime="6.270963372s" totalInitContainerRuntime="21.364664ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:04.899948315 +0000 UTC m=+58.952920288" watchObservedRunningTime="2026-10-31 00:30:04.901394314 +0000 UTC m=+58.954366290" Oct 31 00:30:06.612398 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:06.612366 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.306874 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:07.306841 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.324734 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:07.324708 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.903176 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:07.903148 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:11.148059 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.148017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:30:11.148466 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.148077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:30:11.150609 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.150590 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:30:11.151489 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.151473 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:30:11.160873 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.160847 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:30:11.161006 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.160958 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:30:11.249258 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.249220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkk68\" (UniqueName: \"kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68\") pod \"network-check-target-nsk8c\" (UID: \"698ab510-f0c2-4745-b410-b1697d1aabb8\") " pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:30:11.251802 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.251783 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:30:11.261907 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.261891 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:30:11.272453 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.272423 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkk68\" (UniqueName: \"kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68\") pod \"network-check-target-nsk8c\" (UID: \"698ab510-f0c2-4745-b410-b1697d1aabb8\") " pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:30:11.416572 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.416553 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9g68m\"" Oct 31 00:30:11.420745 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.420729 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-brqzn\"" Oct 31 00:30:11.424842 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.424829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:30:11.428784 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.428754 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:30:11.430796 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.430780 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xkpl6\"" Oct 31 00:30:11.439378 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.439362 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:30:11.703946 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.703920 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87jmt"] Oct 31 00:30:11.707258 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.707226 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wj84l"] Oct 31 00:30:11.712866 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:30:11.712841 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d4d8b4_ee1c_4703_8ac5_f739d72ab13d.slice/crio-cae181bc57daaf41938b81bd72e8ad0d42d9609cf014453d1f00d8e6a28aa964 WatchSource:0}: Error finding container cae181bc57daaf41938b81bd72e8ad0d42d9609cf014453d1f00d8e6a28aa964: Status 404 returned error can't find the container with id cae181bc57daaf41938b81bd72e8ad0d42d9609cf014453d1f00d8e6a28aa964 Oct 31 00:30:11.727218 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.727199 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nsk8c"] Oct 31 00:30:11.729141 ip-10-0-134-252 kubenswrapper[2576]: W1031 00:30:11.729108 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698ab510_f0c2_4745_b410_b1697d1aabb8.slice/crio-5271ade19a81c446f0f994f97672f219fedd6a245f19eec3dfaa38a47c173f42 WatchSource:0}: Error finding container 5271ade19a81c446f0f994f97672f219fedd6a245f19eec3dfaa38a47c173f42: Status 404 returned error can't find the container with id 5271ade19a81c446f0f994f97672f219fedd6a245f19eec3dfaa38a47c173f42 Oct 31 00:30:11.897687 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.897643 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wj84l" event={"ID":"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d","Type":"ContainerStarted","Data":"cae181bc57daaf41938b81bd72e8ad0d42d9609cf014453d1f00d8e6a28aa964"} Oct 31 00:30:11.898725 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.898704 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nsk8c" event={"ID":"698ab510-f0c2-4745-b410-b1697d1aabb8","Type":"ContainerStarted","Data":"5271ade19a81c446f0f994f97672f219fedd6a245f19eec3dfaa38a47c173f42"} Oct 31 00:30:11.899748 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:11.899730 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87jmt" event={"ID":"f4b96af0-063c-4752-a290-12371f09d52a","Type":"ContainerStarted","Data":"89b861d91a88a6ad4d3cdb59f72cd06c04bbf54ef8ac5a35dd98c4a3e7ae7567"} Oct 31 00:30:13.911174 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:13.911108 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87jmt" event={"ID":"f4b96af0-063c-4752-a290-12371f09d52a","Type":"ContainerStarted","Data":"185ee88462c03cfd543e8ce113c1ab1f57ef8432918d6f60f7681f1801e7a6e2"} Oct 31 00:30:13.911174 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:13.911172 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87jmt" event={"ID":"f4b96af0-063c-4752-a290-12371f09d52a","Type":"ContainerStarted","Data":"b4926365eb60f82f958686b1ed3d86780a846a076562223d73716573aab73ea4"} Oct 31 00:30:13.928182 ip-10-0-134-252 kubenswrapper[2576]: I1031 00:30:13.928140 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-87jmt" podStartSLOduration=65.823751849 podStartE2EDuration="1m6.928128568s" totalImagesPullingTime="1.104376719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:13.92785522 +0000 UTC m=+67.980827195" watchObservedRunningTime="2026-10-31 00:30:13.928128568 +0000 UTC m=+67.981100535" Oct 31 00:30:16.561346 ip-10-0-134-252 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:30:16.580180 ip-10-0-134-252 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:30:16.580381 ip-10-0-134-252 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:30:16.580576 ip-10-0-134-252 systemd[1]: kubelet.service: Consumed 3.966s CPU time. Oct 31 00:30:16.593539 ip-10-0-134-252 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:30:16.677096 ip-10-0-134-252 kubenswrapper[5821]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:30:16.677096 ip-10-0-134-252 kubenswrapper[5821]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:30:16.677096 ip-10-0-134-252 kubenswrapper[5821]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:30:16.677096 ip-10-0-134-252 kubenswrapper[5821]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:30:16.679450 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679357 5821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:30:16.679450 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679448 5821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:30:16.679450 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679454 5821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:30:16.679450 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679459 5821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679462 5821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679466 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679469 5821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679471 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679474 5821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679477 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679480 5821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679482 5821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679485 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679487 5821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679496 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679499 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679501 5821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679504 5821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679507 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679510 5821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679513 5821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679516 5821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:30:16.679676 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679518 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679521 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679524 5821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679526 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679530 5821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679534 5821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679537 5821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679540 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679543 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679547 5821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679552 5821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679556 5821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679560 5821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679565 5821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679569 5821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679573 5821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679577 5821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679581 5821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679585 5821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:30:16.680310 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679590 5821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679593 5821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679596 5821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679599 5821 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679602 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679605 5821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679608 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679611 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679613 5821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679631 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679635 5821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679637 5821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679640 5821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679642 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679645 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679647 5821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679650 5821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679653 5821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679655 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679658 5821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:30:16.680953 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679660 5821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679663 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679665 5821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679668 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679671 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679673 5821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679679 5821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679682 5821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679685 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679688 5821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679690 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679693 5821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679696 5821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679698 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679701 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679703 5821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679706 5821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679708 5821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679711 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:30:16.681740 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679714 5821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679716 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679719 5821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679722 5821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679724 5821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679726 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679729 5821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679731 5821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679734 5821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679736 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679740 5821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679744 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679749 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679753 5821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679760 5821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679766 5821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679770 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679774 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679778 5821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679785 5821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:30:16.682337 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679789 5821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679792 5821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679794 5821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679797 5821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679800 5821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.679802 5821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680806 5821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680814 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680817 5821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680820 5821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680823 5821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680825 5821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680828 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680838 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680840 5821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680843 5821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680846 5821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680849 5821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680852 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:30:16.682971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680854 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680859 5821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680863 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680866 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680869 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680872 5821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680874 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680877 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680879 5821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680882 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680884 5821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680887 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680891 5821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680895 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680898 5821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680900 5821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680903 5821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680905 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680908 5821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:30:16.683552 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680912 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680916 5821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680921 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680926 5821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680932 5821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680937 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680942 5821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680946 5821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680961 5821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680966 5821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680970 5821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680976 5821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680980 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680982 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680985 5821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680988 5821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680990 5821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680993 5821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680995 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:30:16.684162 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.680998 5821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681000 5821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681003 5821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681005 5821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681008 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681010 5821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681031 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681035 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681038 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681041 5821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681043 5821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681046 5821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681049 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681051 5821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681054 5821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681056 5821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681059 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681061 5821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681064 5821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681066 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:30:16.684839 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681069 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681071 5821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681079 5821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681082 5821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681084 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681087 5821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681089 5821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681092 5821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681094 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681096 5821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681099 5821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681102 5821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681104 5821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681107 5821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681110 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681112 5821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681116 5821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681118 5821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681121 5821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681124 5821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:30:16.685483 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681126 5821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681129 5821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681131 5821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681134 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681138 5821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681143 5821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681147 5821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681151 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681154 5821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681158 5821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681162 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681165 5821 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681169 5821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681173 5821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681177 5821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681804 5821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681812 5821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681816 5821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681818 5821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681821 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:30:16.686079 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681824 5821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681827 5821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681830 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681832 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681835 5821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681838 5821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681840 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681843 5821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681846 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681848 5821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681850 5821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681854 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681856 5821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681859 5821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681862 5821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681864 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681876 5821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681881 5821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681884 5821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681886 5821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:30:16.686751 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681889 5821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681892 5821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681894 5821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681897 5821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681900 5821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681902 5821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681904 5821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681908 5821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681911 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681913 5821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681916 5821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681920 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681924 5821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681931 5821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681935 5821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681939 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681943 5821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681946 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681950 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:30:16.687604 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681954 5821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681958 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681962 5821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681966 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681971 5821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681974 5821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681976 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681979 5821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681981 5821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681984 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681986 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681989 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681991 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681994 5821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681996 5821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.681999 5821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682001 5821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682004 5821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682006 5821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682008 5821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:30:16.688282 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682011 5821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682014 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682017 5821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682019 5821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682022 5821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682024 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682028 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682031 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682033 5821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682036 5821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682038 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682041 5821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682044 5821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682046 5821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682049 5821 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682052 5821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682056 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682059 5821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682061 5821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:30:16.688913 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682064 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682066 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682069 5821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682072 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682074 5821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682076 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682079 5821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682081 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682085 5821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682089 5821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682094 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682099 5821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682103 5821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682107 5821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682111 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682115 5821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682119 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682123 5821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682127 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:30:16.689461 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682133 5821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682137 5821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682139 5821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682142 5821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682252 5821 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682264 5821 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682272 5821 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682279 5821 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682286 5821 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682290 5821 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682301 5821 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682308 5821 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682313 5821 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682317 5821 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682322 5821 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682328 5821 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682332 5821 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682337 5821 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682341 5821 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682344 5821 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682347 5821 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682350 5821 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682355 5821 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682358 5821 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:30:16.690100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682361 5821 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682364 5821 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682367 5821 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682371 5821 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682374 5821 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682378 5821 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682381 5821 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682384 5821 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682389 5821 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682391 5821 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682394 5821 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682398 5821 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682402 5821 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682405 5821 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682408 5821 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682410 5821 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682413 5821 flags.go:64] FLAG: --enable-server="true" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682416 5821 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682422 5821 flags.go:64] FLAG: --event-burst="100" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682425 5821 flags.go:64] FLAG: --event-qps="50" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682428 5821 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682431 5821 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682436 5821 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682443 5821 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:30:16.690801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682448 5821 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682452 5821 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682456 5821 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682461 5821 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682465 5821 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682470 5821 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682474 5821 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682479 5821 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682483 5821 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682485 5821 flags.go:64] FLAG: --feature-gates="" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682489 5821 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682492 5821 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682496 5821 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682499 5821 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682502 5821 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682506 5821 flags.go:64] FLAG: --help="false" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682510 5821 flags.go:64] FLAG: --hostname-override="ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682513 5821 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682516 5821 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682519 5821 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682522 5821 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682525 5821 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682528 5821 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:30:16.691518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682531 5821 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682534 5821 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682537 5821 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682540 5821 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682543 5821 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682546 5821 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682549 5821 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682551 5821 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682554 5821 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682557 5821 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682560 5821 flags.go:64] FLAG: --lock-file="" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682563 5821 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682566 5821 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682568 5821 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682574 5821 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682577 5821 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682580 5821 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682583 5821 flags.go:64] FLAG: --logging-format="text" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682586 5821 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682589 5821 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682592 5821 flags.go:64] FLAG: --manifest-url="" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682595 5821 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682599 5821 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682603 5821 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682611 5821 flags.go:64] FLAG: --max-pods="110" Oct 31 00:30:16.692338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682632 5821 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682638 5821 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682642 5821 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682647 5821 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682651 5821 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682656 5821 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682660 5821 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682669 5821 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682672 5821 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682676 5821 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682679 5821 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682682 5821 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682684 5821 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682687 5821 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682690 5821 flags.go:64] FLAG: --port="10250" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682693 5821 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682696 5821 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0684ee449620158ad" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682699 5821 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682702 5821 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682705 5821 flags.go:64] FLAG: --register-node="true" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682708 5821 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682711 5821 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682714 5821 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682717 5821 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682720 5821 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682724 5821 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:30:16.693071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682727 5821 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682730 5821 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682732 5821 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682735 5821 flags.go:64] FLAG: --runonce="false" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682739 5821 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682741 5821 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682747 5821 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682750 5821 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682753 5821 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682756 5821 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682759 5821 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682762 5821 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682765 5821 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682768 5821 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682771 5821 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682775 5821 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682780 5821 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682785 5821 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682789 5821 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682797 5821 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682801 5821 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682805 5821 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682811 5821 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682816 5821 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682821 5821 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:30:16.694011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682824 5821 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682827 5821 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682830 5821 flags.go:64] FLAG: --v="2" Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682834 5821 flags.go:64] FLAG: --version="false" Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682838 5821 flags.go:64] FLAG: --vmodule="" Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682842 5821 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.682845 5821 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.682999 5821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683004 5821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683007 5821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683010 5821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683012 5821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683015 5821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683019 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683022 5821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683025 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683028 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683031 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683033 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683036 5821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683038 5821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:30:16.694743 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683040 5821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683045 5821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683048 5821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683051 5821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683054 5821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683058 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683063 5821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683067 5821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683071 5821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683074 5821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683078 5821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683082 5821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683086 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683090 5821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683094 5821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683098 5821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683102 5821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683105 5821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683109 5821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683113 5821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:30:16.695400 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683117 5821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683119 5821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683122 5821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683125 5821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683130 5821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683132 5821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683136 5821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683138 5821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683141 5821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683144 5821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683146 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683149 5821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683151 5821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683155 5821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683158 5821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683160 5821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683163 5821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683165 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683167 5821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683170 5821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:30:16.696019 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683172 5821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683175 5821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683177 5821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683180 5821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683183 5821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683186 5821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683189 5821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683192 5821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683195 5821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683197 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683200 5821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683202 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683205 5821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683207 5821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683210 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683214 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683220 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683224 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683228 5821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:30:16.696673 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683232 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683236 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683240 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683244 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683248 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683251 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683257 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683261 5821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683264 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683267 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683271 5821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683275 5821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683279 5821 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683283 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683288 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683291 5821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683293 5821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683296 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683298 5821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:30:16.697463 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683301 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683303 5821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683306 5821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683308 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683311 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683313 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683316 5821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683318 5821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683321 5821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683323 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683327 5821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683330 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683332 5821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683335 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.683371 5821 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683420 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:30:16.698304 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683425 5821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683427 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683430 5821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683435 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683437 5821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683440 5821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683442 5821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683445 5821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683448 5821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683450 5821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683453 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683455 5821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683458 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683461 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683463 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683466 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683468 5821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683470 5821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683473 5821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683476 5821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:30:16.698945 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683480 5821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683484 5821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683489 5821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683493 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683497 5821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683500 5821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683504 5821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683510 5821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683514 5821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683518 5821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683523 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683526 5821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683529 5821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683531 5821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683534 5821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683538 5821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683542 5821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683545 5821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683548 5821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:30:16.699869 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683551 5821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683554 5821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683556 5821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683559 5821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683562 5821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683564 5821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683567 5821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683570 5821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683572 5821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683574 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683577 5821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683580 5821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683582 5821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683585 5821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683587 5821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683589 5821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683592 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683595 5821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683597 5821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683599 5821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:30:16.700505 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683604 5821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683606 5821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683609 5821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683631 5821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683637 5821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683642 5821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683646 5821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683650 5821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683657 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683661 5821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683665 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683669 5821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683673 5821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683676 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683678 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683681 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683683 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683686 5821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683688 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:30:16.701287 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683691 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683694 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683696 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683699 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683701 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683704 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683707 5821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683709 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683711 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683714 5821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683716 5821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683719 5821 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683721 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683727 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683730 5821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683732 5821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683735 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683737 5821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683740 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:30:16.701909 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683743 5821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683747 5821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683750 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683753 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683755 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683758 5821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683760 5821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.683763 5821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.689788 5821 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.689809 5821 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.689835 5821 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.689843 5821 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.689940 5821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.689949 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.689956 5821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.689963 5821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.689969 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.689973 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.689978 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:30:16.702440 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.689982 5821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.689986 5821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.689990 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.689995 5821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690000 5821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690004 5821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690008 5821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690012 5821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690016 5821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690020 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690024 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690028 5821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690033 5821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690037 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690042 5821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690047 5821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690051 5821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690055 5821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690060 5821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690065 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:30:16.703013 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690069 5821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690073 5821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690077 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690081 5821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690085 5821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690089 5821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690093 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690097 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690101 5821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690105 5821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690110 5821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690114 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690118 5821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690122 5821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690126 5821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690130 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690134 5821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690138 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690142 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690154 5821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:30:16.703578 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690158 5821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690162 5821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690167 5821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690171 5821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690175 5821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690179 5821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690183 5821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690188 5821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690192 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690196 5821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690200 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690204 5821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690209 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690213 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690217 5821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690221 5821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690226 5821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690230 5821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690235 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:30:16.704349 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690239 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690243 5821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690247 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690251 5821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690258 5821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690264 5821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690269 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690274 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690278 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690282 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690286 5821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690290 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690294 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690305 5821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690309 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690313 5821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690317 5821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690321 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690325 5821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:30:16.705052 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690329 5821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690333 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690337 5821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690341 5821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690346 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690350 5821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690354 5821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690358 5821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690362 5821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690366 5821 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690370 5821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690375 5821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690379 5821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690382 5821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690386 5821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690390 5821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690394 5821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690399 5821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690403 5821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690406 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:30:16.705688 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690410 5821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.690455 5821 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690695 5821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690702 5821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690707 5821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690711 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690715 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690723 5821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690728 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690733 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690737 5821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690741 5821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690746 5821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690750 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690754 5821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690758 5821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:30:16.706406 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690762 5821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690766 5821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690773 5821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690779 5821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690784 5821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690788 5821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690792 5821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690796 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690800 5821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690804 5821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690809 5821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690814 5821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690818 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690822 5821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690827 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690831 5821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690835 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690839 5821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690842 5821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:30:16.707043 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690846 5821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690850 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690854 5821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690858 5821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690862 5821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690866 5821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690870 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690875 5821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690879 5821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690883 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690887 5821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690892 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690896 5821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690900 5821 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690904 5821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690908 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690912 5821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690916 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690919 5821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:30:16.707865 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690923 5821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690927 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690931 5821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690935 5821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690939 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690943 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690947 5821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690951 5821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690955 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690959 5821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690963 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690966 5821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690970 5821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690974 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690978 5821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690982 5821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690985 5821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690989 5821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690993 5821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.690999 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:30:16.708539 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691003 5821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691007 5821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691011 5821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691015 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691019 5821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691023 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691028 5821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691032 5821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691036 5821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691040 5821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691044 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691048 5821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691053 5821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691057 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691061 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691065 5821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691069 5821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691073 5821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691077 5821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691081 5821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:30:16.709246 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691085 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691089 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691093 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691097 5821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691101 5821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691105 5821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691109 5821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691113 5821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691117 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691121 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691125 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691129 5821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691134 5821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691138 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.691174 5821 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691223 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:30:16.709907 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691230 5821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691235 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691240 5821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691245 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691249 5821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691253 5821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691257 5821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691262 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691266 5821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691269 5821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691274 5821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691277 5821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691281 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691286 5821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691290 5821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691294 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691299 5821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691303 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691308 5821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691312 5821 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:30:16.710513 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691316 5821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691320 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691325 5821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691328 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691332 5821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691336 5821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691340 5821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691344 5821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691348 5821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691353 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691358 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691362 5821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691366 5821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691370 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691374 5821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691378 5821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691382 5821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691385 5821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691389 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691393 5821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:30:16.711360 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691397 5821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691401 5821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691405 5821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691408 5821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691412 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691416 5821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691420 5821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691424 5821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691428 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691432 5821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691436 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691442 5821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691446 5821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691450 5821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691454 5821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691458 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691462 5821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691467 5821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691471 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691475 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:30:16.712199 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691479 5821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691483 5821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691488 5821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691492 5821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691498 5821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691502 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691506 5821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691510 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691514 5821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691518 5821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691522 5821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691526 5821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691530 5821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691534 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691539 5821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691543 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691547 5821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691551 5821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691555 5821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691559 5821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:30:16.713179 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691563 5821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691567 5821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691571 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691575 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691581 5821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691587 5821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691592 5821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691596 5821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691601 5821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691605 5821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691608 5821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691613 5821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691631 5821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691636 5821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691640 5821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691649 5821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691654 5821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691659 5821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691664 5821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:30:16.714078 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691668 5821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691672 5821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691676 5821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691679 5821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691683 5821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:16.691687 5821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.691769 5821 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.693284 5821 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.693358 5821 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.693634 5821 server.go:1014] "Starting client certificate rotation" Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.693738 5821 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.693811 5821 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:24:06 +0000 UTC" deadline="2028-08-02 19:12:25.173710294 +0000 UTC" Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.693851 5821 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15402h42m8.479862608s" Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.694344 5821 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.696045 5821 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.697910 5821 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:30:16.714902 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.700242 5821 log.go:25] "Validated CRI v1 image API" Oct 31 00:30:16.715559 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.701035 5821 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:30:16.715559 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.703350 5821 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a9400b02-3866-42d7-8b09-596edb3239a5:/dev/nvme0n1p4 acdf1e7a-0579-4ba1-bd25-8939952ea356:/dev/nvme0n1p3] Oct 31 00:30:16.715700 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.703373 5821 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0551584673ab68d38c82443ba50bfd39f610979e1c4fa42ddd15024a346d0cd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0551584673ab68d38c82443ba50bfd39f610979e1c4fa42ddd15024a346d0cd8/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fc2d4cf6ec9212d61c8d76159bea8753442415ba2429ea12898622feec41bb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fc2d4cf6ec9212d61c8d76159bea8753442415ba2429ea12898622feec41bb9/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19c09337655a903b6c9d5a935ef64772f2f4f3241dc488fd30039fc4b2b91e6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19c09337655a903b6c9d5a935ef64772f2f4f3241dc488fd30039fc4b2b91e6d/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33e287d9a9d90eea698b1d49ef45eb8ecc3edf7c15489120309e505a674fea5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33e287d9a9d90eea698b1d49ef45eb8ecc3edf7c15489120309e505a674fea5e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5271ade19a81c446f0f994f97672f219fedd6a245f19eec3dfaa38a47c173f42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5271ade19a81c446f0f994f97672f219fedd6a245f19eec3dfaa38a47c173f42/userdata/shm major:0 minor:379 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/583b9b18ba6ebc6dad4e716877a8995f997a662b6214e4cbabdfa9e9744447ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/583b9b18ba6ebc6dad4e716877a8995f997a662b6214e4cbabdfa9e9744447ca/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/623f37df70210507f4c3f43133c9802b978f52ccc4269ddb66a60d04898f6dbd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/623f37df70210507f4c3f43133c9802b978f52ccc4269ddb66a60d04898f6dbd/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66ae9e7e3ff66cbb21d9c557544e24d5c884109124fc1560576160869237dc2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66ae9e7e3ff66cbb21d9c557544e24d5c884109124fc1560576160869237dc2c/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70f761047a9fe5bb8263a0f28a78c32a17d695df1f2cac4cc9ffbfa9d240207f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70f761047a9fe5bb8263a0f28a78c32a17d695df1f2cac4cc9ffbfa9d240207f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8240f10e9131e2cc38e21c7d60715c22b3de5d759ac772ba3a459b5970804f42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8240f10e9131e2cc38e21c7d60715c22b3de5d759ac772ba3a459b5970804f42/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89b861d91a88a6ad4d3cdb59f72cd06c04bbf54ef8ac5a35dd98c4a3e7ae7567/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89b861d91a88a6ad4d3cdb59f72cd06c04bbf54ef8ac5a35dd98c4a3e7ae7567/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d965f84be745e3e58f3e0d386b962fca2c6e255395faaef2fc50f2c18a6cd52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d965f84be745e3e58f3e0d386b962fca2c6e255395faaef2fc50f2c18a6cd52/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d4b6c1858c21c1556fccb4d0402efd73216d33f91e8ddfb133d21de4cf52d0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d4b6c1858c21c1556fccb4d0402efd73216d33f91e8ddfb133d21de4cf52d0c/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f9d03935951e80f18e6c1b2e8f8463fbcb619f90c4624966a71dd7182c89180/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f9d03935951e80f18e6c1b2e8f8463fbcb619f90c4624966a71dd7182c89180/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b765ff17367f70849cc916487c1410916d39fd9c8054300bcf050cec9c75dcf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b765ff17367f70849cc916487c1410916d39fd9c8054300bcf050cec9c75dcf2/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cae181bc57daaf41938b81bd72e8ad0d42d9609cf014453d1f00d8e6a28aa964/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cae181bc57daaf41938b81bd72e8ad0d42d9609cf014453d1f00d8e6a28aa964/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0a3acbc83a5f38728133e3dfa5b0ea2507f076b332b8da29358f27740a86c2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0a3acbc83a5f38728133e3dfa5b0ea2507f076b332b8da29358f27740a86c2f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6d0ab3c41a935419164bdaf1be1cd9d21b29cd0fda71550a04de8f9c9ca6cc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6d0ab3c41a935419164bdaf1be1cd9d21b29cd0fda71550a04de8f9c9ca6cc3/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7c94a13db71ce4c24ca57fa25974d9ac751d012664ea32f2f27b146deeeeca6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7c94a13db71ce4c24ca57fa25974d9ac751d012664ea32f2f27b146deeeeca6/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcbd30c9a52d9925e349b62c93ae7b8f7e6377aeca2b79dc7f6bea32cc9a8918/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcbd30c9a52d9925e349b62c93ae7b8f7e6377aeca2b79dc7f6bea32cc9a8918/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd51a42ccb70e635a0e592c6baa68e6cb37b6bec9723874747a07385d7d40bf3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd51a42ccb70e635a0e592c6baa68e6cb37b6bec9723874747a07385d7d40bf3/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/089a2aaa-c90a-4df7-8e97-09650f16aab4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/089a2aaa-c90a-4df7-8e97-09650f16aab4/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b02c695-144a-4481-bc14-1cd89baba762/volumes/kubernetes.io~projected/kube-api-access-bf2x6:{mountpoint:/var/lib/kubelet/pods/0b02c695-144a-4481-bc14-1cd89baba762/volumes/kubernetes.io~projected/kube-api-access-bf2x6 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volume-subpaths/web-config/config-reloader/2 major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volume-subpaths/web-config/prometheus/15 major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~empty-dir/config-out major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~projected/kube-api-access-vppsr:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~projected/kube-api-access-vppsr major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~projected/tls-assets major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/config major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/web-config major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/226527e6-3023-4c5a-b600-fb6322b77877/volumes/kubernetes.io~projected/kube-api-access-dwjn5:{mountpoint:/var/lib/kubelet/pods/226527e6-3023-4c5a-b600-fb6322b77877/volumes/kubernetes.io~projected/kube-api-access-dwjn5 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/226527e6-3023-4c5a-b600-fb6322b77877/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/226527e6-3023-4c5a-b600-fb6322b77877/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33c29ec2-dcd1-4c23-99cc-66c6684db03a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/33c29ec2-dcd1-4c23-99cc-66c6684db03a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33c29ec2-dcd1-4c23-99cc-66c6684db03a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/33c29ec2-dcd1-4c23-99cc-66c6684db03a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33c29ec2-dcd1-4c23-99cc-66c6684db03a/volumes/kubernetes.io~projected/kube-api-access-4967k:{mountpoint:/var/lib/kubelet/pods/33c29ec2-dcd1-4c23-99cc-66c6684db03a/volumes/kubernetes.io~projected/kube-api-access-4967k major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50ef0d54-5212-427c-8bab-04b4262c3e11/volumes/kubernetes.io~projected/kube-api-access-xgndg:{mountpoint:/var/lib/kubelet/pods/50ef0d54-5212-427c-8bab-04b4262c3e11/volumes/kubernetes.io~projected/kube-api-access-xgndg major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/658d1734-2e40-4784-8c85-d57ce4c5c7a8/volumes/kubernetes.io~projected/kube-api-access-lndm5:{mountpoint:/var/lib/kubelet/pods/658d1734-2e40-4784-8c85-d57ce4c5c7a8/volumes/kubernetes.io~projected/kube-api-access-lndm5 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/698ab510-f0c2-4745-b410-b1697d1aabb8/volumes/kubernetes.io~projected/kube-api-access-jkk68:{mountpoint:/var/lib/kubelet/pods/698ab510-f0c2-4745-b410-b1697d1aabb8/volumes/kubernetes.io~projected/kube-api-access-jkk68 major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/912d5cf1-39a7-4fe3-8e72-d6b20bd67476/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/912d5cf1-39a7-4fe3-8e72-d6b20bd67476/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/912d5cf1-39a7-4fe3-8e72-d6b20bd67476/volumes/kubernetes.io~projected/kube-api-access-jcgv9:{mountpoint:/var/lib/kubelet/pods/912d5cf1-39a7-4fe3-8e72-d6b20bd67476/volumes/kubernetes.io~projected/kube-api-access-jcgv9 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/912d5cf1-39a7-4fe3-8e72-d6b20bd67476/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/912d5cf1-39a7-4fe3-8e72-d6b20bd67476/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97194d65-9d7e-47b2-8c55-2283301e8a4d/volumes/kubernetes.io~projected/kube-api-access-qfvq9:{mountpoint:/var/lib/kubelet/pods/97194d65-9d7e-47b2-8c55-2283301e8a4d/volumes/kubernetes.io~projected/kube-api-access-qfvq9 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97194d65-9d7e-47b2-8c55-2283301e8a4d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/97194d65-9d7e-47b2-8c55-2283301e8a4d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e/volumes/kubernetes.io~projected/kube-api-access-gjz8r:{mountpoint:/var/lib/kubelet/pods/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e/volumes/kubernetes.io~projected/kube-api-access-gjz8r major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b22fa0b3-c20e-402b-81b5-898a838d42c9/volumes/kubernetes.io~projected/kube-api-access-pq7sf:{mountpoint:/var/lib/kubelet/pods/b22fa0b3-c20e-402b-81b5-898a838d42c9/volumes/kubernetes.io~projected/kube-api-access-pq7sf major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b22fa0b3-c20e-402b-81b5-898a838d42c9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b22fa0b3-c20e-402b-81b5-898a838d42c9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b22fa0b3-c20e-402b-81b5-898a838d42c9/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/b22fa0b3-c20e-402b-81b5-898a838d42c9/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db73288c-815d-49ed-9cdb-6799fb6ecc7b/volumes/kubernetes.io~projected/kube-api-access-bpqqj:{mountpoint:/var/lib/kubelet/pods/db73288c-815d-49ed-9cdb-6799fb6ecc7b/volumes/kubernetes.io~projected/kube-api-access-bpqqj major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2acbfb-3c5b-4d40-8365-066e0d29f72e/volumes/kubernetes.io~projected/kube-api-access-zmrdz:{mountpoint:/var/lib/kubelet/pods/ea2acbfb-3c5b-4d40-8365-066e0d29f72e/volumes/kubernetes.io~projected/kube-api-access-zmrdz major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2acbfb-3c5b-4d40-8365-066e0d29f72e/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ea2acbfb-3c5b-4d40-8365-066e0d29f72e/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea2acbfb-3c5b-4d40-8365-066e0d29f72e/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ea2acbfb-3c5b-4d40-8365-066e0d29f72e/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4b96af0-063c-4752-a290-12371f09d52a/volumes/kubernetes.io~projected/kube-api-access-zkkh4:{mountpoint:/var/lib/kubelet/pods/f4b96af0-063c-4752-a290-12371f09d52a/volumes/kubernetes.io~projected/kube-api-access-zkkh4 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4b96af0-063c-4752-a290-12371f09d52a/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f4b96af0-063c-4752-a290-12371f09d52a/volumes/kubernetes.io~secret/metrics-certs major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f79572d1-5d21-4771-b613-85a421e550b3/volumes/kubernetes.io~projected/kube-api-access-64kwx:{mountpoint:/var/lib/kubelet/pods/f79572d1-5d21-4771-b613-85a421e550b3/volumes/kubernetes.io~projected/kube-api-access-64kwx major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f79572d1-5d21-4771-b613-85a421e550b3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/f79572d1-5d21-4771-b613-85a421e550b3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb2daade-8f58-42e8-a79b-2a373ff186ea/volumes/kubernetes.io~projected/kube-api-access-vszj2:{mountpoint:/var/lib/kubelet/pods/fb2daade-8f58-42e8-a79b-2a373ff186ea/volumes/kubernetes.io~projected/kube-api-access-vszj2 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fba123f2-dd49-407a-8e1d-c568c0d66ce4/volumes/kubernetes.io~projected/kube-api-access-v24jd:{mountpoint:/var/lib/kubelet/pods/fba123f2-dd49-407a-8e1d-c568c0d66ce4/volumes/kubernetes.io~projected/kube-api-access-v24jd major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fba123f2-dd49-407a-8e1d-c568c0d66ce4/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fba123f2-dd49-407a-8e1d-c568c0d66ce4/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fba123f2-dd49-407a-8e1d-c568c0d66ce4/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/fba123f2-dd49-407a-8e1d-c568c0d66ce4/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522/volumes/kubernetes.io~projected/kube-api-access-87hzr:{mountpoint:/var/lib/kubelet/pods/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522/volumes/kubernetes.io~projected/kube-api-access-87hzr major:0 minor:52 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0f093fd3f7206a5435c29ffa1264b1755227baa1258f3d2b68f7461cf1c2c005/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4c3a56d6fb6a5b1d30cebb67a87c152c92459fe6a4610141ff57270fedc0f738/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/cec06146afeaadd33f5f3038df21cf1d53d8f850d2215085205bc3f9c447eb5c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/1b40977b59d7ff6fcf97ea9d1b28d70c3c609f05869db1858cdf2f570a0fd86e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/4badfa6f6c42ec0572d79921e5732cd221a03c92928273ac24dd8e8fc7505483/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-118:{mountpoint:/var/lib/containers/storage/overlay/c1b8b1ecd09462d3692dfe8d1523c1850f62a8bc3b649b543ab0b9e0e8c5df13/merged major:0 minor:118 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/27b3a35370ecadb0ae6ac4ca211e4e8b8ed2ed19474d93be094039432b5416c3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/99cd13a08e70eb72a83f0421603300cdba43b59da8728926f6c789ac4ac09ef3/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3bb62d37ca7d3140332545f128bc1d04ec2ca13620de3fbced93046acde56797/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/f68a8add595eb3e18d192a5b33a6d278b5d1150d004d2d7acaeebbc7961e8738/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/bcfb0a112609ff4301bcbb936481427bc30782edd03e9c6487cb7c927411a16f/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/43603e996c1d6b70b3fbaeed87e92567d74a7a5f8a0701b6c5dd25ccf6cd8a5c/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/966cc10970af29220780516f315f36bc8a5f1d59619c5517890e1b19e0efcda7/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/87152aed748242994b7d9c8f4fa3536967137aafbff7fbdf243d5a9f3f1594b3/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-155:{mountpoint:/var/lib/containers/storage/overlay/ecd16e7c76b6185f551900d7c0a8cf3a4e02d800e6ccdb8cebc341ebb4bb7d4a/merged major:0 minor:155 fsType:overlay blockSize:0} overlay_0-160:{mountpoint:/var/lib/containers/storage/overlay/4fd7f544f1f57196b1723ba3c84c05869784298951b2bae9555ca2fe6f4baed5/merged major:0 minor:160 fsType:overlay blockSize:0} overlay_0-165:{mountpoint:/var/lib/containers/storage/overlay/433a4142a612a1740272afd65e4c8c2e9686c26eaa626d2744ef62cd64f6d7db/merged major:0 minor:165 fsType:overlay blockSize:0} overlay_0-170:{mountpoint:/var/lib/containers/storage/overlay/86db05c5509e3694779dcb1c61a4dfb5527144e3dbd6f4615f81707328ae51b7/merged major:0 minor:170 fsType:overlay blockSize:0} overlay_0-175:{mountpoint:/var/lib/containers/storage/overlay/9b12ee6865568b0ee7c57454d3bdb2563878f6ffbdbe3e08a69cd26d3e686b4d/merged major:0 minor:175 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5826c2cbaa12691f72de58f0250fc1e4c97c6948c8a526d5d73b4d085ef23a4b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7cdad28e25d5c129dbf285b8896e14fedb009697b29a6fd38b01533cb654b826/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/dbf5cd04a684ce7d2f3df6e89392d57d49c77cd481c54e849d9b7238b0748163/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/241cbe8a6cd13703595b6b70aacb777688db270f4338cb29394c1e84ee36033f/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/3d718de8f28c54b0e554bf329550e396809247bd63205136230bad4bd15dd70e/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/dc6213ec8f3fb5a3ad39669a51be2ea4a84759efa3744ff2b84b74893df31240/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/de72e2090a09a2806476ba37c44871eb18e16c69ec8de230a5b6a58820d560f1/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/3a94e5c01986fb087ac2b2d8bcbc235ee352a499566bbebd354d83a9b24fa167/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/3381cbfde9012b4c83fa7ca8fa219f3a09754dab959f5d9ca1e30c61bd36f296/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/c3bb25817b2206f031bc9200924aeed06466a43c0d1524556f62013893536547/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/d86caf46f4263a26e7eecfe1cebd87fe6f27229f9503fb0b1333f17c63360266/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/5d9df67049d02a8d15992c14c9a9332b88b3f8d3b37c259b7efc5e8b11a48b7b/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/396db771a358269d1b2d7af7496cec630ac99fc88e5124bd7de87d8c55c75fb7/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/256b7c49a3549aebf5646d21d18e638455cd99f46355a9a069b404f060b13f2c/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/59572303fbbfafdbb2983fa9eda39eb0186a139d9ae54a4b78b42e34e9a17970/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/52253f1073be87e2acb6a1a97ddcf96e817de2abb7012ddd311217f6a6ddfd7a/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/3d5c487acafa19a76404c28b097c1b065a8eb25e002dfd32d2397cb7cdff205f/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/ac7736cf573a15d1c92dfd92e9dc6cc56711e2aefacf0d8ee8f09a812ffc8027/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/7a350e0908099bd6f552eaa6403a36292e17e602e1627184992b43ea5f5c7cdd/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/00f13a8858ac7a2715c3d54642720c16296effcfd4bb401ceefb0158b9164a71/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/71a3af789165f3bb63553d2cadd724a54acae89a17f3f9ac7c42b8b19783613d/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/404fe7eae3ea42ef2fd64405eb0ecc922beadad0b851cdd7901bc4ecd38feabf/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/0715e0d81718fc72342ed973f877ff5fb936c2722c5a6bf8b16adf6131b4d096/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/ae206d207e3a699ea119bab497f0af85accbf8a7bfa923509cf02033daf58cf2/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/bdb2747e5a4645d08ae166b31b5b85ff5242ebce8ff4b3a124895324f816b162/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/9c28bb265a1e365191888e978bc0c20088f9232a3b3c6e25a9e995c1cd238cca/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/dd4114724547925c065f2d00fa7c6c1e4e83c1ada3bd1c8fce136607fe59ae0f/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/ea6b2e696b82a3274e6c965b794a278ebe769d0bc44756df51e89c6bfd8a69f5/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/b88c2288f893f7832ae4feabfb05ede80ad79529207ad1d874cb5567d0f00d6b/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/72a75d0b035e7227658141ffbefce53efcc3258a1edc56100068fdf90389e55d/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/4f72114124ee638136720f643e20b213cc1fa9093014aaa804c3e7be79b3d192/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/921fdb2391c0aed19f418ad1e2fd35a887b6bc21d319297367fd9f65fd5d7cfb/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/35aeaf1908cd58b221db78273e25c5e0ac53b91fe3aab1a2e4aadbd786a9c4e8/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/98a82fff321a790877b5c73317138cf338460db8ccef852e090a837eee03cbe0/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/72d2ab917dde69bdf7733bf3e541a16dd6ff230bd7d60bf77216951ecf4d9517/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/c62ec62a88665c385758d5dff6ab687956a36381c684f68bdf84d2c2eb57c51c/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/cfb9e92fd4c0ada417874fb023ec2033a96d5cf75a07681e75c68531b2d7bed1/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/5b2f30791216f7d5b12d3c26e7530e82f598558146143185c065bac1c3a46a00/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/dc2184a99fc74015049ab0580d3431c89c6e9871bb6b19b6cadade273395d934/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/203db1938cca0f31674caaebcff93333bfa4a20c7cc9fbc198631f93a70acb11/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/74f238dd5ae46dc7cdeb6c128a523f9ba9ec1c3594ee62d2b881377431521348/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/22031d5f39ce768411def835b9ad41c00f9e9c7832f8b8a07e1e6e533111743b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/6bdb103e140d814bc7f8193cd1b8c24d26c2ba4b249c7d621ff522fb27db6712/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/1630e7697153ad22065e8f60b6bfacefd13574ee43a35beafecf57affc43bbfa/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/94bb9ee6bb26ff234aa8a0de7b5f6498229e817cfe87f1d5b456983695d19ae0/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ee5996d5e7a12a295ba5e32847f39abf2ec38b89206a6b217e2214c51472fa25/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/1248052ae68154d85e26b828c1f66bc68a4ef9872b1be98fbe1ac476f1f7d76a/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.713766 5821 manager.go:217] Machine: {Timestamp:2026-10-31 00:30:16.713140904 +0000 UTC m=+0.074723533 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100096 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23d3eb8aa50bad25f7e995d2cbc58c SystemUUID:ec23d3eb-8aa5-0bad-25f7-e995d2cbc58c BootID:5fcc6b60-02cc-4d59-ac0f-95b702b9f0c5 Filesystems:[{Device:/var/lib/kubelet/pods/ea2acbfb-3c5b-4d40-8365-066e0d29f72e/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fba123f2-dd49-407a-8e1d-c568c0d66ce4/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66ae9e7e3ff66cbb21d9c557544e24d5c884109124fc1560576160869237dc2c/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2acbfb-3c5b-4d40-8365-066e0d29f72e/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/912d5cf1-39a7-4fe3-8e72-d6b20bd67476/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/912d5cf1-39a7-4fe3-8e72-d6b20bd67476/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b765ff17367f70849cc916487c1410916d39fd9c8054300bcf050cec9c75dcf2/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:321 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:323 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f9d03935951e80f18e6c1b2e8f8463fbcb619f90c4624966a71dd7182c89180/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~projected/kube-api-access-vppsr DeviceMajor:0 DeviceMinor:330 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/623f37df70210507f4c3f43133c9802b978f52ccc4269ddb66a60d04898f6dbd/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fba123f2-dd49-407a-8e1d-c568c0d66ce4/volumes/kubernetes.io~projected/kube-api-access-v24jd DeviceMajor:0 DeviceMinor:245 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b22fa0b3-c20e-402b-81b5-898a838d42c9/volumes/kubernetes.io~projected/kube-api-access-pq7sf DeviceMajor:0 DeviceMinor:246 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:319 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:324 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:310 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:323 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4b96af0-063c-4752-a290-12371f09d52a/volumes/kubernetes.io~projected/kube-api-access-zkkh4 DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/089a2aaa-c90a-4df7-8e97-09650f16aab4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-160 DeviceMajor:0 DeviceMinor:160 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b22fa0b3-c20e-402b-81b5-898a838d42c9/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:244 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/698ab510-f0c2-4745-b410-b1697d1aabb8/volumes/kubernetes.io~projected/kube-api-access-jkk68 DeviceMajor:0 DeviceMinor:374 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/33c29ec2-dcd1-4c23-99cc-66c6684db03a/volumes/kubernetes.io~projected/kube-api-access-4967k DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea2acbfb-3c5b-4d40-8365-066e0d29f72e/volumes/kubernetes.io~projected/kube-api-access-zmrdz DeviceMajor:0 DeviceMinor:247 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19c09337655a903b6c9d5a935ef64772f2f4f3241dc488fd30039fc4b2b91e6d/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50ef0d54-5212-427c-8bab-04b4262c3e11/volumes/kubernetes.io~projected/kube-api-access-xgndg DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fb2daade-8f58-42e8-a79b-2a373ff186ea/volumes/kubernetes.io~projected/kube-api-access-vszj2 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0b02c695-144a-4481-bc14-1cd89baba762/volumes/kubernetes.io~projected/kube-api-access-bf2x6 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:320 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:323 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/db73288c-815d-49ed-9cdb-6799fb6ecc7b/volumes/kubernetes.io~projected/kube-api-access-bpqqj DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d965f84be745e3e58f3e0d386b962fca2c6e255395faaef2fc50f2c18a6cd52/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-170 DeviceMajor:0 DeviceMinor:170 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f79572d1-5d21-4771-b613-85a421e550b3/volumes/kubernetes.io~projected/kube-api-access-64kwx DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/658d1734-2e40-4784-8c85-d57ce4c5c7a8/volumes/kubernetes.io~projected/kube-api-access-lndm5 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:329 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-165 DeviceMajor:0 DeviceMinor:165 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d4b6c1858c21c1556fccb4d0402efd73216d33f91e8ddfb133d21de4cf52d0c/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b22fa0b3-c20e-402b-81b5-898a838d42c9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33c29ec2-dcd1-4c23-99cc-66c6684db03a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fba123f2-dd49-407a-8e1d-c568c0d66ce4/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:242 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcbd30c9a52d9925e349b62c93ae7b8f7e6377aeca2b79dc7f6bea32cc9a8918/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/912d5cf1-39a7-4fe3-8e72-d6b20bd67476/volumes/kubernetes.io~projected/kube-api-access-jcgv9 DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-118 DeviceMajor:0 DeviceMinor:118 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-155 DeviceMajor:0 DeviceMinor:155 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8240f10e9131e2cc38e21c7d60715c22b3de5d759ac772ba3a459b5970804f42/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0551584673ab68d38c82443ba50bfd39f610979e1c4fa42ddd15024a346d0cd8/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-175 DeviceMajor:0 DeviceMinor:175 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:322 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4b96af0-063c-4752-a290-12371f09d52a/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:369 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89b861d91a88a6ad4d3cdb59f72cd06c04bbf54ef8ac5a35dd98c4a3e7ae7567/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33c29ec2-dcd1-4c23-99cc-66c6684db03a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:325 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/583b9b18ba6ebc6dad4e716877a8995f997a662b6214e4cbabdfa9e9744447ca/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97194d65-9d7e-47b2-8c55-2283301e8a4d/volumes/kubernetes.io~projected/kube-api-access-qfvq9 DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7c94a13db71ce4c24ca57fa25974d9ac751d012664ea32f2f27b146deeeeca6/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:323 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fc2d4cf6ec9212d61c8d76159bea8753442415ba2429ea12898622feec41bb9/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/97194d65-9d7e-47b2-8c55-2283301e8a4d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5271ade19a81c446f0f994f97672f219fedd6a245f19eec3dfaa38a47c173f42/userdata/shm DeviceMajor:0 DeviceMinor:379 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/226527e6-3023-4c5a-b600-fb6322b77877/volumes/kubernetes.io~projected/kube-api-access-dwjn5 DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:373 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0a3acbc83a5f38728133e3dfa5b0ea2507f076b332b8da29358f27740a86c2f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/226527e6-3023-4c5a-b600-fb6322b77877/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e/volumes/kubernetes.io~projected/kube-api-access-gjz8r DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:326 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522/volumes/kubernetes.io~projected/kube-api-access-87hzr DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd51a42ccb70e635a0e592c6baa68e6cb37b6bec9723874747a07385d7d40bf3/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33e287d9a9d90eea698b1d49ef45eb8ecc3edf7c15489120309e505a674fea5e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6d0ab3c41a935419164bdaf1be1cd9d21b29cd0fda71550a04de8f9c9ca6cc3/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f79572d1-5d21-4771-b613-85a421e550b3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70f761047a9fe5bb8263a0f28a78c32a17d695df1f2cac4cc9ffbfa9d240207f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cae181bc57daaf41938b81bd72e8ad0d42d9609cf014453d1f00d8e6a28aa964/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:5271ade19a81c44 MacAddress:62:f9:41:d6:df:04 Speed:10000 Mtu:8901} {Name:623f37df7021050 MacAddress:be:27:a6:79:13:f0 Speed:10000 Mtu:8901} {Name:66ae9e7e3ff66cb MacAddress:82:eb:9f:e2:25:75 Speed:10000 Mtu:8901} {Name:8240f10e9131e2c MacAddress:f6:cf:0a:90:0f:9c Speed:10000 Mtu:8901} {Name:89b861d91a88a6a MacAddress:e2:01:c9:45:62:31 Speed:10000 Mtu:8901} {Name:9f9d03935951e80 MacAddress:2e:d3:de:3a:02:17 Speed:10000 Mtu:8901} {Name:b765ff17367f708 MacAddress:a6:f3:82:6a:c3:5b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:df:da:f0:a3 Speed:0 Mtu:9001} {Name:br-int MacAddress:2e:cb:5f:1c:f2:6f Speed:0 Mtu:8901} {Name:cae181bc57daaf4 MacAddress:4a:8c:5e:ec:d1:ec Speed:10000 Mtu:8901} {Name:d7c94a13db71ce4 MacAddress:5a:17:e4:b8:b0:98 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:df:da:f0:a3 Speed:0 Mtu:9001} {Name:fcbd30c9a52d992 MacAddress:a2:04:2a:98:99:57 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:c2:b2:09:cc:52:c4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5e:89:a6:bb:6a:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.714254 5821 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.714311 5821 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.714647 5821 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.714675 5821 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-252.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.714829 5821 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.714837 5821 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.714848 5821 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.714867 5821 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.714964 5821 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.715148 5821 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.715274 5821 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.715309 5821 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.715325 5821 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.715338 5821 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.715352 5821 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:30:16.716056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.715363 5821 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:30:16.716958 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.716206 5821 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:30:16.716958 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.716219 5821 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:30:16.717809 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.717791 5821 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:30:16.726317 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726295 5821 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:30:16.726317 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726316 5821 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:30:16.726436 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726422 5821 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:30:16.726679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726654 5821 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:30:16.726736 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726685 5821 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:30:16.726801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726789 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:30:16.726860 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726810 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:30:16.726860 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726820 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:30:16.726860 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726830 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:30:16.726860 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726839 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:30:16.726860 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726848 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:30:16.726860 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726857 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:30:16.727100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726866 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:30:16.727100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726878 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:30:16.727100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726887 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:30:16.727100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726899 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:30:16.727100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726908 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:30:16.727100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726931 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:30:16.727100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.726937 5821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:30:16.731786 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.731200 5821 server.go:1263] "Started kubelet" Oct 31 00:30:16.731786 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.731487 5821 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:30:16.731786 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.731561 5821 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:30:16.731786 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.731691 5821 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:30:16.732291 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.732269 5821 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:30:16.732439 ip-10-0-134-252 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:30:16.738661 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.734358 5821 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:30:16.738661 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.735390 5821 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:30:16.738661 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.737407 5821 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:30:16.738661 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.737486 5821 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:30:16.738661 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.737548 5821 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:24:12 +0000 UTC" deadline="2028-04-03 13:01:28.92490548 +0000 UTC" Oct 31 00:30:16.738661 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.737563 5821 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:30:16.738661 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.737566 5821 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12492h31m12.187342559s" Oct 31 00:30:16.738661 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.737575 5821 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:30:16.738661 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.737604 5821 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:30:16.738661 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.737704 5821 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:30:16.740017 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.739969 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="912d5cf1-39a7-4fe3-8e72-d6b20bd67476" volumeName="kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-env-overrides" seLinuxMountContext="" Oct 31 00:30:16.740142 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740020 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2acbfb-3c5b-4d40-8365-066e0d29f72e" volumeName="kubernetes.io/secret/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 00:30:16.740142 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740036 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db73288c-815d-49ed-9cdb-6799fb6ecc7b" volumeName="kubernetes.io/configmap/db73288c-815d-49ed-9cdb-6799fb6ecc7b-cni-binary-copy" seLinuxMountContext="" Oct 31 00:30:16.740142 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740051 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:30:16.740142 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740064 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-metrics-client-certs" seLinuxMountContext="" Oct 31 00:30:16.740142 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740077 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="658d1734-2e40-4784-8c85-d57ce4c5c7a8" volumeName="kubernetes.io/projected/658d1734-2e40-4784-8c85-d57ce4c5c7a8-kube-api-access-lndm5" seLinuxMountContext="" Oct 31 00:30:16.740142 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740089 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b96af0-063c-4752-a290-12371f09d52a" volumeName="kubernetes.io/projected/f4b96af0-063c-4752-a290-12371f09d52a-kube-api-access-zkkh4" seLinuxMountContext="" Oct 31 00:30:16.740142 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740101 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 31 00:30:16.740142 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740116 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-kube-api-access-vppsr" seLinuxMountContext="" Oct 31 00:30:16.740142 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740129 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740153 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-web-config" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740167 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d4d8b4-ee1c-4703-8ac5-f739d72ab13d" volumeName="kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740179 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79572d1-5d21-4771-b613-85a421e550b3" volumeName="kubernetes.io/configmap/f79572d1-5d21-4771-b613-85a421e550b3-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740192 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="089a2aaa-c90a-4df7-8e97-09650f16aab4" volumeName="kubernetes.io/configmap/089a2aaa-c90a-4df7-8e97-09650f16aab4-konnectivity-ca" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740206 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config-out" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740219 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22fa0b3-c20e-402b-81b5-898a838d42c9" volumeName="kubernetes.io/projected/b22fa0b3-c20e-402b-81b5-898a838d42c9-kube-api-access-pq7sf" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740232 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb2daade-8f58-42e8-a79b-2a373ff186ea" volumeName="kubernetes.io/configmap/fb2daade-8f58-42e8-a79b-2a373ff186ea-serviceca" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740243 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba123f2-dd49-407a-8e1d-c568c0d66ce4" volumeName="kubernetes.io/empty-dir/fba123f2-dd49-407a-8e1d-c568c0d66ce4-volume-directive-shadow" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740255 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522" volumeName="kubernetes.io/projected/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-kube-api-access-87hzr" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740271 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740285 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ef0d54-5212-427c-8bab-04b4262c3e11" volumeName="kubernetes.io/projected/50ef0d54-5212-427c-8bab-04b4262c3e11-kube-api-access-xgndg" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740297 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="912d5cf1-39a7-4fe3-8e72-d6b20bd67476" volumeName="kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovnkube-config" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740309 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb2daade-8f58-42e8-a79b-2a373ff186ea" volumeName="kubernetes.io/projected/fb2daade-8f58-42e8-a79b-2a373ff186ea-kube-api-access-vszj2" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740348 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740362 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740375 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="226527e6-3023-4c5a-b600-fb6322b77877" volumeName="kubernetes.io/secret/226527e6-3023-4c5a-b600-fb6322b77877-cert" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740387 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ef0d54-5212-427c-8bab-04b4262c3e11" volumeName="kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740400 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97194d65-9d7e-47b2-8c55-2283301e8a4d" volumeName="kubernetes.io/secret/97194d65-9d7e-47b2-8c55-2283301e8a4d-metrics-tls" seLinuxMountContext="" Oct 31 00:30:16.740416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740420 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22fa0b3-c20e-402b-81b5-898a838d42c9" volumeName="kubernetes.io/configmap/b22fa0b3-c20e-402b-81b5-898a838d42c9-metrics-client-ca" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740434 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79572d1-5d21-4771-b613-85a421e550b3" volumeName="kubernetes.io/empty-dir/f79572d1-5d21-4771-b613-85a421e550b3-data-volume" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740448 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba123f2-dd49-407a-8e1d-c568c0d66ce4" volumeName="kubernetes.io/configmap/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740460 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b02c695-144a-4481-bc14-1cd89baba762" volumeName="kubernetes.io/empty-dir/0b02c695-144a-4481-bc14-1cd89baba762-tmp-dir" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740485 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ef0d54-5212-427c-8bab-04b4262c3e11" volumeName="kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740497 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="912d5cf1-39a7-4fe3-8e72-d6b20bd67476" volumeName="kubernetes.io/projected/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-kube-api-access-jcgv9" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740508 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2acbfb-3c5b-4d40-8365-066e0d29f72e" volumeName="kubernetes.io/configmap/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-metrics-client-ca" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740521 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba123f2-dd49-407a-8e1d-c568c0d66ce4" volumeName="kubernetes.io/secret/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740533 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="912d5cf1-39a7-4fe3-8e72-d6b20bd67476" volumeName="kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740544 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97194d65-9d7e-47b2-8c55-2283301e8a4d" volumeName="kubernetes.io/empty-dir/97194d65-9d7e-47b2-8c55-2283301e8a4d-tmp-dir" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740554 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd77f3f-3824-44dd-8d5f-156ff99d5b7e" volumeName="kubernetes.io/secret/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e-monitoring-plugin-cert" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740567 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="089a2aaa-c90a-4df7-8e97-09650f16aab4" volumeName="kubernetes.io/secret/089a2aaa-c90a-4df7-8e97-09650f16aab4-agent-certs" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740578 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740590 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22fa0b3-c20e-402b-81b5-898a838d42c9" volumeName="kubernetes.io/secret/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740601 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-db" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740613 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-grpc-tls" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740644 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33c29ec2-dcd1-4c23-99cc-66c6684db03a" volumeName="kubernetes.io/empty-dir/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-tuned" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740655 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="658d1734-2e40-4784-8c85-d57ce4c5c7a8" volumeName="kubernetes.io/configmap/658d1734-2e40-4784-8c85-d57ce4c5c7a8-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740665 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22fa0b3-c20e-402b-81b5-898a838d42c9" volumeName="kubernetes.io/empty-dir/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-textfile" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740676 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba123f2-dd49-407a-8e1d-c568c0d66ce4" volumeName="kubernetes.io/projected/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-api-access-v24jd" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740687 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22fa0b3-c20e-402b-81b5-898a838d42c9" volumeName="kubernetes.io/configmap/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740698 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba123f2-dd49-407a-8e1d-c568c0d66ce4" volumeName="kubernetes.io/secret/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740709 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b02c695-144a-4481-bc14-1cd89baba762" volumeName="kubernetes.io/projected/0b02c695-144a-4481-bc14-1cd89baba762-kube-api-access-bf2x6" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740727 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="912d5cf1-39a7-4fe3-8e72-d6b20bd67476" volumeName="kubernetes.io/secret/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740739 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97194d65-9d7e-47b2-8c55-2283301e8a4d" volumeName="kubernetes.io/configmap/97194d65-9d7e-47b2-8c55-2283301e8a4d-config-volume" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740749 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd77f3f-3824-44dd-8d5f-156ff99d5b7e" volumeName="kubernetes.io/projected/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e-kube-api-access-gjz8r" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740756 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db73288c-815d-49ed-9cdb-6799fb6ecc7b" volumeName="kubernetes.io/projected/db73288c-815d-49ed-9cdb-6799fb6ecc7b-kube-api-access-bpqqj" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740764 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b96af0-063c-4752-a290-12371f09d52a" volumeName="kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740777 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79572d1-5d21-4771-b613-85a421e550b3" volumeName="kubernetes.io/secret/f79572d1-5d21-4771-b613-85a421e550b3-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740786 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740794 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33c29ec2-dcd1-4c23-99cc-66c6684db03a" volumeName="kubernetes.io/empty-dir/33c29ec2-dcd1-4c23-99cc-66c6684db03a-tmp" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740801 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2acbfb-3c5b-4d40-8365-066e0d29f72e" volumeName="kubernetes.io/projected/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-kube-api-access-zmrdz" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740809 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79572d1-5d21-4771-b613-85a421e550b3" volumeName="kubernetes.io/projected/f79572d1-5d21-4771-b613-85a421e550b3-kube-api-access-64kwx" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740817 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33c29ec2-dcd1-4c23-99cc-66c6684db03a" volumeName="kubernetes.io/projected/33c29ec2-dcd1-4c23-99cc-66c6684db03a-kube-api-access-4967k" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740828 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740839 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ef0d54-5212-427c-8bab-04b4262c3e11" volumeName="kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-cni-binary-copy" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740847 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97194d65-9d7e-47b2-8c55-2283301e8a4d" volumeName="kubernetes.io/projected/97194d65-9d7e-47b2-8c55-2283301e8a4d-kube-api-access-qfvq9" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740854 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22fa0b3-c20e-402b-81b5-898a838d42c9" volumeName="kubernetes.io/secret/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-tls" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740861 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba123f2-dd49-407a-8e1d-c568c0d66ce4" volumeName="kubernetes.io/configmap/fba123f2-dd49-407a-8e1d-c568c0d66ce4-metrics-client-ca" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740869 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-metrics-client-ca" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740877 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-tls-assets" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740884 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740895 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="226527e6-3023-4c5a-b600-fb6322b77877" volumeName="kubernetes.io/projected/226527e6-3023-4c5a-b600-fb6322b77877-kube-api-access-dwjn5" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740911 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="698ab510-f0c2-4745-b410-b1697d1aabb8" volumeName="kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740922 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db73288c-815d-49ed-9cdb-6799fb6ecc7b" volumeName="kubernetes.io/configmap/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-daemon-config" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740934 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea2acbfb-3c5b-4d40-8365-066e0d29f72e" volumeName="kubernetes.io/secret/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740945 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740958 5821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" volumeName="kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740969 5821 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:30:16.741061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.740976 5821 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:30:16.743013 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.741258 5821 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:30:16.743013 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.741579 5821 factory.go:55] Registering systemd factory Oct 31 00:30:16.743013 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.741596 5821 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:30:16.743013 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.741858 5821 factory.go:153] Registering CRI-O factory Oct 31 00:30:16.743013 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.741915 5821 factory.go:223] Registration of the crio container factory successfully Oct 31 00:30:16.743013 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.741972 5821 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:30:16.743013 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.741992 5821 factory.go:103] Registering Raw factory Oct 31 00:30:16.743013 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.742006 5821 manager.go:1199] Started watching for new ooms in manager Oct 31 00:30:16.743013 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.742302 5821 manager.go:319] Starting recovery of all containers Oct 31 00:30:16.750089 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.748483 5821 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:30:16.770739 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.770532 5821 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:30:16.772196 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.772171 5821 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:30:16.772196 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.772197 5821 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:30:16.772353 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.772234 5821 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:30:16.772353 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:16.772285 5821 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:30:16.778983 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.778857 5821 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:30:16.789984 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.789924 5821 generic.go:356] "Generic (PLEG): container finished" podID="b22fa0b3-c20e-402b-81b5-898a838d42c9" containerID="bc1ab3ca6b68767a8c6a31b043cd01174f78b66193fffbb522774d30014450ff" exitCode=0 Oct 31 00:30:16.801591 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.801568 5821 generic.go:356] "Generic (PLEG): container finished" podID="50ef0d54-5212-427c-8bab-04b4262c3e11" containerID="f1d51eb145745778d1e747fdeb2a811092f425e822389c9eaee01d3a67032164" exitCode=0 Oct 31 00:30:16.801591 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.801589 5821 generic.go:356] "Generic (PLEG): container finished" podID="50ef0d54-5212-427c-8bab-04b4262c3e11" containerID="1a59a881970d973991f963d9fa7d98f9c100c25b23ee421f92a4d2302e70eb3f" exitCode=0 Oct 31 00:30:16.801772 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.801598 5821 generic.go:356] "Generic (PLEG): container finished" podID="50ef0d54-5212-427c-8bab-04b4262c3e11" containerID="4f469a9b112e981dd4d46e9f76e40ec0026e235cb17ae824ce588062eacfcb68" exitCode=0 Oct 31 00:30:16.801772 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.801605 5821 generic.go:356] "Generic (PLEG): container finished" podID="50ef0d54-5212-427c-8bab-04b4262c3e11" containerID="28d59981121f426170fc967c89ee4842380442b1a0360fb6821da00022c4c40c" exitCode=0 Oct 31 00:30:16.801772 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.801613 5821 generic.go:356] "Generic (PLEG): container finished" podID="50ef0d54-5212-427c-8bab-04b4262c3e11" containerID="76063ad4ae8f09d7a6b060bc21392dba7b145f095ae3a3bf85767b99f8afcde0" exitCode=0 Oct 31 00:30:16.801772 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.801636 5821 generic.go:356] "Generic (PLEG): container finished" podID="50ef0d54-5212-427c-8bab-04b4262c3e11" containerID="b410a730df7fd813c062d9c68a9399dc24c0923497cf9cb36636f634037cf0a5" exitCode=0 Oct 31 00:30:16.806216 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.806167 5821 generic.go:356] "Generic (PLEG): container finished" podID="8f3794bd171ae09db1b2f1a9b17e1c85" containerID="dea3b69ae61fd5b8535d5b6417863ac284adb59204d641d3e97e47ca6d45495f" exitCode=0 Oct 31 00:30:16.826391 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.826334 5821 generic.go:356] "Generic (PLEG): container finished" podID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerID="3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263" exitCode=0 Oct 31 00:30:16.853112 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.853090 5821 manager.go:324] Recovery completed Oct 31 00:30:16.873393 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:16.873353 5821 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 00:30:16.879146 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879124 5821 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:30:16.879146 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879143 5821 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:30:16.879346 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879164 5821 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:30:16.879346 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879319 5821 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:30:16.879346 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879332 5821 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:30:16.879478 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879353 5821 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:30:16.879478 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879361 5821 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:30:16.879478 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879368 5821 policy_none.go:50] "Start" Oct 31 00:30:16.879478 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879379 5821 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:30:16.879478 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879391 5821 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:30:16.879713 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879520 5821 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:30:16.879713 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879527 5821 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:30:16.879713 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.879533 5821 policy_none.go:44] "Start" Oct 31 00:30:16.885981 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.885354 5821 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:30:16.890110 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.890094 5821 manager.go:341] "Starting Device Plugin manager" Oct 31 00:30:16.890217 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:16.890127 5821 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:30:16.890217 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.890139 5821 server.go:87] "Starting device plugin registration server" Oct 31 00:30:16.890479 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.890460 5821 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:30:16.890574 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.890480 5821 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:30:16.890688 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.890677 5821 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:30:16.890785 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.890756 5821 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:30:16.890838 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.890793 5821 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:30:16.890838 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.890801 5821 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:30:16.890940 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.890861 5821 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:30:16.890782826Z","UUID":"be8864d5-59a0-4d3c-bcec-6d04a43aa997","Handler":null,"Name":"","Endpoint":""} Oct 31 00:30:16.892201 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.892174 5821 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:30:16.892279 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.892211 5821 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:30:16.892901 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:16.892876 5821 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:30:16.990833 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.990796 5821 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:30:16.993132 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.993109 5821 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:30:16.993257 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.993150 5821 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:30:16.993257 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.993168 5821 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-252.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:30:16.993257 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:16.993227 5821 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.007703 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.007683 5821 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.007826 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.007748 5821 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.075930 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.075855 5821 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal"] Oct 31 00:30:17.076683 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.076569 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" event={"ID":"8f3794bd171ae09db1b2f1a9b17e1c85","Type":"ContainerStarted","Data":"52c191c962653ed7e8820d03343a000bea366bb3294568a9ff03869a1eae8a24"} Oct 31 00:30:17.076683 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.076650 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" event={"ID":"8f3794bd171ae09db1b2f1a9b17e1c85","Type":"ContainerDied","Data":"dea3b69ae61fd5b8535d5b6417863ac284adb59204d641d3e97e47ca6d45495f"} Oct 31 00:30:17.076683 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.076668 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" event={"ID":"8f3794bd171ae09db1b2f1a9b17e1c85","Type":"ContainerStarted","Data":"9d4b6c1858c21c1556fccb4d0402efd73216d33f91e8ddfb133d21de4cf52d0c"} Oct 31 00:30:17.076899 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.076740 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" event={"ID":"703d740fe3de84816febee5a3376c36a","Type":"ContainerStarted","Data":"aa51ff09c002cd72f5a0ca4d92fd7b6ef2d5f802af72f3f7fbec94889fa645bc"} Oct 31 00:30:17.076899 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.076758 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" event={"ID":"703d740fe3de84816febee5a3376c36a","Type":"ContainerStarted","Data":"fd51a42ccb70e635a0e592c6baa68e6cb37b6bec9723874747a07385d7d40bf3"} Oct 31 00:30:17.077155 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.077038 5821 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.077521 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.077497 5821 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.088923 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.088905 5821 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:30:17.089041 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:17.088960 5821 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.089692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.089667 5821 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:30:17.089791 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:17.089712 5821 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.149969 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.149932 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f3794bd171ae09db1b2f1a9b17e1c85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"8f3794bd171ae09db1b2f1a9b17e1c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.149969 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.149970 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/703d740fe3de84816febee5a3376c36a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"703d740fe3de84816febee5a3376c36a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.150178 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.149987 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8f3794bd171ae09db1b2f1a9b17e1c85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"8f3794bd171ae09db1b2f1a9b17e1c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.250576 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.250529 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/703d740fe3de84816febee5a3376c36a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"703d740fe3de84816febee5a3376c36a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.250576 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.250578 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8f3794bd171ae09db1b2f1a9b17e1c85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"8f3794bd171ae09db1b2f1a9b17e1c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.250842 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.250602 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f3794bd171ae09db1b2f1a9b17e1c85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"8f3794bd171ae09db1b2f1a9b17e1c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.250842 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.250671 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f3794bd171ae09db1b2f1a9b17e1c85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"8f3794bd171ae09db1b2f1a9b17e1c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.250842 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.250680 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/703d740fe3de84816febee5a3376c36a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"703d740fe3de84816febee5a3376c36a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.250842 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.250738 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8f3794bd171ae09db1b2f1a9b17e1c85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal\" (UID: \"8f3794bd171ae09db1b2f1a9b17e1c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal" Oct 31 00:30:17.716933 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.716890 5821 apiserver.go:52] "Watching apiserver" Oct 31 00:30:17.737540 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.737469 5821 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:30:17.740812 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.738902 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-252.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-wj84l","openshift-dns/node-resolver-m8d27","openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb","openshift-network-operator/iptables-alerter-9l4x8","openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm","openshift-ovn-kubernetes/ovnkube-node-swvwl","kube-system/kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal","openshift-insights/insights-runtime-extractor-g62jt","openshift-dns/dns-default-xbdxj","openshift-ingress-canary/ingress-canary-8ssc4","openshift-monitoring/node-exporter-7jm8z","openshift-cluster-node-tuning-operator/tuned-pptjg","openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4","openshift-multus/multus-9zlgx","openshift-network-diagnostics/network-check-target-nsk8c","kube-system/konnectivity-agent-d4jrp","openshift-image-registry/node-ca-bmn4m","openshift-multus/multus-additional-cni-plugins-jqkmw","openshift-monitoring/prometheus-k8s-0","openshift-multus/network-metrics-daemon-87jmt"] Oct 31 00:30:17.747015 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.746985 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:30:17.747133 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.747107 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:30:17.747224 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.747107 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:30:17.747224 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.747178 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pl6tx\"" Oct 31 00:30:17.747224 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.747189 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-l9cjs\"" Oct 31 00:30:17.747404 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.747386 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:30:17.747449 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.747402 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:30:17.747449 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.747423 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:30:17.747527 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.747465 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:30:17.747604 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.747585 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:30:17.747903 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.747883 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:30:17.747990 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.747902 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:30:17.748256 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748205 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:30:17.748256 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748235 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:30:17.748506 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748476 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:30:17.748567 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748510 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:30:17.748567 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748533 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-brqzn\"" Oct 31 00:30:17.749050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748699 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9g68m\"" Oct 31 00:30:17.749050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748730 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:30:17.749050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748743 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:30:17.749050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748749 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jhx4l\"" Oct 31 00:30:17.749050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748797 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:30:17.749050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748861 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:30:17.749050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748865 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:30:17.749050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748956 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rn9j5\"" Oct 31 00:30:17.749050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.748984 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:30:17.749050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749013 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:30:17.749050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749027 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:30:17.749050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749035 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:30:17.749642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749192 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:30:17.749642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749197 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:30:17.749642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749200 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:30:17.749642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749244 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:30:17.749642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749296 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4cbmj\"" Oct 31 00:30:17.749642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749325 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pnvd5\"" Oct 31 00:30:17.749642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749245 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-72tmz\"" Oct 31 00:30:17.749642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749457 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xkpl6\"" Oct 31 00:30:17.749642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749464 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:30:17.749642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749509 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:30:17.750077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749749 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9nqcj\"" Oct 31 00:30:17.750077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749833 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:30:17.750077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749861 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vxzr5\"" Oct 31 00:30:17.750077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749866 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:30:17.750077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.749941 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:30:17.750077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.750040 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:30:17.750077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.750052 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:30:17.750332 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.750238 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:30:17.750332 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.750261 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:30:17.750332 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.750291 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:30:17.750332 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.750304 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zxj7m\"" Oct 31 00:30:17.750485 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.750398 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:30:17.750485 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.750413 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pcfpx\"" Oct 31 00:30:17.751149 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.751054 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-trzmz\"" Oct 31 00:30:17.751149 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.751112 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:30:17.751149 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.751113 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:30:17.751882 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.751831 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:30:17.752058 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.752041 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z8njm\"" Oct 31 00:30:17.752121 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.752072 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:30:17.752590 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.752571 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:30:17.753068 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.753050 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:30:17.753447 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.753431 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:30:17.753863 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.753846 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gm9sb\"" Oct 31 00:30:17.754025 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.754010 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:30:17.754118 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.754101 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:30:17.754240 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.754227 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:30:17.754319 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.754277 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:30:17.754695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.754674 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:30:17.754794 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.754780 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:30:17.754969 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.754949 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:30:17.755420 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.755400 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kvd4r\"" Oct 31 00:30:17.755521 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.755508 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-c87xz\"" Oct 31 00:30:17.756652 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.756613 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:30:17.757390 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.757366 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:30:17.757526 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.757492 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:30:17.757720 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.757694 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:30:17.757809 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.757727 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jgktj\"" Oct 31 00:30:17.757809 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.757739 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:30:17.757809 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.757746 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:30:17.757809 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.757766 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:30:17.758116 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.758097 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:30:17.758214 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.758198 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:30:17.758277 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.758265 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:30:17.758379 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.758303 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fmft110ekd91k\"" Oct 31 00:30:17.758379 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.758318 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:30:17.758379 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.758332 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:30:17.758379 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.758337 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:30:17.758504 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.758381 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:30:17.759698 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.759684 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:30:17.840175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.840146 5821 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:30:17.853149 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853096 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-lib-modules\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.853149 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853126 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.853149 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853153 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-run-ovn-kubernetes\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.853442 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853181 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-env-overrides\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.853442 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853229 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vszj2\" (UniqueName: \"kubernetes.io/projected/fb2daade-8f58-42e8-a79b-2a373ff186ea-kube-api-access-vszj2\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:30:17.853442 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853276 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.853442 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853346 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:30:17.853654 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853453 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-root\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.853654 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853481 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-env-overrides\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.853654 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853491 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.853654 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853545 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-systemd-units\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.853654 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853576 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-cni-netd\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.853654 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853575 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:30:17.853654 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853597 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.853654 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853641 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-slash\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853669 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853687 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-cni-multus\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853707 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-kubelet\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853746 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-var-lib-kubelet\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853772 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-cnibin\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853810 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853794 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db73288c-815d-49ed-9cdb-6799fb6ecc7b-cni-binary-copy\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853841 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64kwx\" (UniqueName: \"kubernetes.io/projected/f79572d1-5d21-4771-b613-85a421e550b3-kube-api-access-64kwx\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853860 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853882 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v24jd\" (UniqueName: \"kubernetes.io/projected/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-api-access-v24jd\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853908 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853909 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rz5vm\" (UID: \"9cd77f3f-3824-44dd-8d5f-156ff99d5b7e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:30:17.854070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.853936 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854083 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854130 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rz5vm\" (UID: \"9cd77f3f-3824-44dd-8d5f-156ff99d5b7e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854173 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-sys\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854200 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db73288c-815d-49ed-9cdb-6799fb6ecc7b-cni-binary-copy\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854203 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fba123f2-dd49-407a-8e1d-c568c0d66ce4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854256 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854283 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854308 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-socket-dir-parent\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854335 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854365 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854390 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fba123f2-dd49-407a-8e1d-c568c0d66ce4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854395 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-kubelet\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854492 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854517 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-k8s-cni-cncf-io\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854543 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-etc-kubernetes\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854573 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b96af0-063c-4752-a290-12371f09d52a-metrics-certs\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854591 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f79572d1-5d21-4771-b613-85a421e550b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854658 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-wtmp\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.854689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854686 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-var-lib-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854711 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-socket-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854737 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-device-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854744 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-original-pull-secret\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854750 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854764 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97194d65-9d7e-47b2-8c55-2283301e8a4d-metrics-tls\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854793 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-run-netns\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854826 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-tuned\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854856 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f79572d1-5d21-4771-b613-85a421e550b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854881 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854906 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854932 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854960 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854972 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f79572d1-5d21-4771-b613-85a421e550b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.854985 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-node-log\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855027 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855039 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97194d65-9d7e-47b2-8c55-2283301e8a4d-metrics-tls\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855105 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-tuned\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855139 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-sys\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855170 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855196 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855231 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-systemd\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855297 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855308 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f79572d1-5d21-4771-b613-85a421e550b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855429 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855480 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855201 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855570 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.855648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855640 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-hostroot\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855669 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855703 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-sys-fs\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855737 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4967k\" (UniqueName: \"kubernetes.io/projected/33c29ec2-dcd1-4c23-99cc-66c6684db03a-kube-api-access-4967k\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855753 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855771 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndm5\" (UniqueName: \"kubernetes.io/projected/658d1734-2e40-4784-8c85-d57ce4c5c7a8-kube-api-access-lndm5\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855830 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysconfig\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855935 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmrdz\" (UniqueName: \"kubernetes.io/projected/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-kube-api-access-zmrdz\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.855961 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856007 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-modprobe-d\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856109 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-systemd\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856137 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856138 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/089a2aaa-c90a-4df7-8e97-09650f16aab4-agent-certs\") pod \"konnectivity-agent-d4jrp\" (UID: \"089a2aaa-c90a-4df7-8e97-09650f16aab4\") " pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856205 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b22fa0b3-c20e-402b-81b5-898a838d42c9-metrics-client-ca\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856232 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-log-socket\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856257 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovn-node-metrics-cert\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856282 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87hzr\" (UniqueName: \"kubernetes.io/projected/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-kube-api-access-87hzr\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856308 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/658d1734-2e40-4784-8c85-d57ce4c5c7a8-iptables-alerter-script\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856332 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/089a2aaa-c90a-4df7-8e97-09650f16aab4-konnectivity-ca\") pod \"konnectivity-agent-d4jrp\" (UID: \"089a2aaa-c90a-4df7-8e97-09650f16aab4\") " pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856363 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97194d65-9d7e-47b2-8c55-2283301e8a4d-config-volume\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856391 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856405 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/089a2aaa-c90a-4df7-8e97-09650f16aab4-agent-certs\") pod \"konnectivity-agent-d4jrp\" (UID: \"089a2aaa-c90a-4df7-8e97-09650f16aab4\") " pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856417 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjz8r\" (UniqueName: \"kubernetes.io/projected/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e-kube-api-access-gjz8r\") pod \"monitoring-plugin-77f4f76594-rz5vm\" (UID: \"9cd77f3f-3824-44dd-8d5f-156ff99d5b7e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856442 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856466 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f79572d1-5d21-4771-b613-85a421e550b3-crio-socket\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856490 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856515 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856538 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/226527e6-3023-4c5a-b600-fb6322b77877-cert\") pod \"ingress-canary-8ssc4\" (UID: \"226527e6-3023-4c5a-b600-fb6322b77877\") " pod="openshift-ingress-canary/ingress-canary-8ssc4" Oct 31 00:30:17.856575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856565 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856590 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkk68\" (UniqueName: \"kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68\") pod \"network-check-target-nsk8c\" (UID: \"698ab510-f0c2-4745-b410-b1697d1aabb8\") " pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856657 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b02c695-144a-4481-bc14-1cd89baba762-tmp-dir\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856683 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-etc-selinux\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856858 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b22fa0b3-c20e-402b-81b5-898a838d42c9-metrics-client-ca\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.856863 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857073 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/226527e6-3023-4c5a-b600-fb6322b77877-cert\") pod \"ingress-canary-8ssc4\" (UID: \"226527e6-3023-4c5a-b600-fb6322b77877\") " pod="openshift-ingress-canary/ingress-canary-8ssc4" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857092 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovn-node-metrics-cert\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857289 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97194d65-9d7e-47b2-8c55-2283301e8a4d-config-volume\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857289 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857484 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857522 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/658d1734-2e40-4784-8c85-d57ce4c5c7a8-host-slash\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857548 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-netns\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857548 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857569 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b02c695-144a-4481-bc14-1cd89baba762-tmp-dir\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857653 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb2daade-8f58-42e8-a79b-2a373ff186ea-host\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857696 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857740 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-multus-certs\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857764 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/089a2aaa-c90a-4df7-8e97-09650f16aab4-konnectivity-ca\") pod \"konnectivity-agent-d4jrp\" (UID: \"089a2aaa-c90a-4df7-8e97-09650f16aab4\") " pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857781 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857805 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpqqj\" (UniqueName: \"kubernetes.io/projected/db73288c-815d-49ed-9cdb-6799fb6ecc7b-kube-api-access-bpqqj\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857839 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjn5\" (UniqueName: \"kubernetes.io/projected/226527e6-3023-4c5a-b600-fb6322b77877-kube-api-access-dwjn5\") pod \"ingress-canary-8ssc4\" (UID: \"226527e6-3023-4c5a-b600-fb6322b77877\") " pod="openshift-ingress-canary/ingress-canary-8ssc4" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857903 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.857979 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.858073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858022 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-conf-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858137 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858138 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgndg\" (UniqueName: \"kubernetes.io/projected/50ef0d54-5212-427c-8bab-04b4262c3e11-kube-api-access-xgndg\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858190 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858196 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/658d1734-2e40-4784-8c85-d57ce4c5c7a8-iptables-alerter-script\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858309 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-ovn\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858357 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-textfile\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858385 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-tls\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858408 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgv9\" (UniqueName: \"kubernetes.io/projected/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-kube-api-access-jcgv9\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858415 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858445 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33c29ec2-dcd1-4c23-99cc-66c6684db03a-tmp\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858451 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-textfile\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858468 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2x6\" (UniqueName: \"kubernetes.io/projected/0b02c695-144a-4481-bc14-1cd89baba762-kube-api-access-bf2x6\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858504 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858537 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkh4\" (UniqueName: \"kubernetes.io/projected/f4b96af0-063c-4752-a290-12371f09d52a-kube-api-access-zkkh4\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858568 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysctl-conf\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858591 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-run\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858513 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33c29ec2-dcd1-4c23-99cc-66c6684db03a-tmp\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858589 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-tls\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858644 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858763 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858800 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-registration-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858839 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-kubernetes\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858879 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-cni-bin\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858931 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-system-cni-dir\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858962 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-host\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.858993 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-system-cni-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859081 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-cni-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859117 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859148 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfvq9\" (UniqueName: \"kubernetes.io/projected/97194d65-9d7e-47b2-8c55-2283301e8a4d-kube-api-access-qfvq9\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859173 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859202 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovnkube-config\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859226 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-kubelet-config\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859250 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-cnibin\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859277 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859303 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97194d65-9d7e-47b2-8c55-2283301e8a4d-tmp-dir\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859303 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.859312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859324 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b02c695-144a-4481-bc14-1cd89baba762-hosts-file\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859348 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7sf\" (UniqueName: \"kubernetes.io/projected/b22fa0b3-c20e-402b-81b5-898a838d42c9-kube-api-access-pq7sf\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859380 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859404 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-daemon-config\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859405 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859468 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f79572d1-5d21-4771-b613-85a421e550b3-data-volume\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859498 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fba123f2-dd49-407a-8e1d-c568c0d66ce4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859544 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppsr\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-kube-api-access-vppsr\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859570 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysctl-d\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859574 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vszj2\" (UniqueName: \"kubernetes.io/projected/fb2daade-8f58-42e8-a79b-2a373ff186ea-kube-api-access-vszj2\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859929 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-dbus\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.860082 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.859604 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-daemon-config\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.860502 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb2daade-8f58-42e8-a79b-2a373ff186ea-serviceca\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.860530 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.860565 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-etc-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.860610 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f79572d1-5d21-4771-b613-85a421e550b3-data-volume\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.860816 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fba123f2-dd49-407a-8e1d-c568c0d66ce4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.861002 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.861104 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovnkube-config\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.861679 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.861579 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v24jd\" (UniqueName: \"kubernetes.io/projected/fba123f2-dd49-407a-8e1d-c568c0d66ce4-kube-api-access-v24jd\") pod \"kube-state-metrics-5dc8d7d89b-rcwt4\" (UID: \"fba123f2-dd49-407a-8e1d-c568c0d66ce4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rcwt4" Oct 31 00:30:17.862729 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.861724 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb2daade-8f58-42e8-a79b-2a373ff186ea-serviceca\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:30:17.862729 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.861877 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lndm5\" (UniqueName: \"kubernetes.io/projected/658d1734-2e40-4784-8c85-d57ce4c5c7a8-kube-api-access-lndm5\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:30:17.862729 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.862578 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmrdz\" (UniqueName: \"kubernetes.io/projected/ea2acbfb-3c5b-4d40-8365-066e0d29f72e-kube-api-access-zmrdz\") pod \"openshift-state-metrics-7bd7489957-nv2qb\" (UID: \"ea2acbfb-3c5b-4d40-8365-066e0d29f72e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nv2qb" Oct 31 00:30:17.862729 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.862709 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97194d65-9d7e-47b2-8c55-2283301e8a4d-tmp-dir\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:30:17.862909 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.862852 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4967k\" (UniqueName: \"kubernetes.io/projected/33c29ec2-dcd1-4c23-99cc-66c6684db03a-kube-api-access-4967k\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.862993 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.862973 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50ef0d54-5212-427c-8bab-04b4262c3e11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.863041 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.863017 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.863112 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.863052 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-cni-bin\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.863112 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.863079 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovnkube-script-lib\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.863244 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.863207 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64kwx\" (UniqueName: \"kubernetes.io/projected/f79572d1-5d21-4771-b613-85a421e550b3-kube-api-access-64kwx\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:30:17.863423 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.863397 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-ovnkube-script-lib\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.863785 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.863680 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.864123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.864102 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjz8r\" (UniqueName: \"kubernetes.io/projected/9cd77f3f-3824-44dd-8d5f-156ff99d5b7e-kube-api-access-gjz8r\") pod \"monitoring-plugin-77f4f76594-rz5vm\" (UID: \"9cd77f3f-3824-44dd-8d5f-156ff99d5b7e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:30:17.864123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.864117 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkk68\" (UniqueName: \"kubernetes.io/projected/698ab510-f0c2-4745-b410-b1697d1aabb8-kube-api-access-jkk68\") pod \"network-check-target-nsk8c\" (UID: \"698ab510-f0c2-4745-b410-b1697d1aabb8\") " pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:30:17.864711 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.864571 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjn5\" (UniqueName: \"kubernetes.io/projected/226527e6-3023-4c5a-b600-fb6322b77877-kube-api-access-dwjn5\") pod \"ingress-canary-8ssc4\" (UID: \"226527e6-3023-4c5a-b600-fb6322b77877\") " pod="openshift-ingress-canary/ingress-canary-8ssc4" Oct 31 00:30:17.864845 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.864750 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87hzr\" (UniqueName: \"kubernetes.io/projected/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-kube-api-access-87hzr\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.864845 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.864792 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgndg\" (UniqueName: \"kubernetes.io/projected/50ef0d54-5212-427c-8bab-04b4262c3e11-kube-api-access-xgndg\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.865065 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.865045 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpqqj\" (UniqueName: \"kubernetes.io/projected/db73288c-815d-49ed-9cdb-6799fb6ecc7b-kube-api-access-bpqqj\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.865349 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.865327 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2x6\" (UniqueName: \"kubernetes.io/projected/0b02c695-144a-4481-bc14-1cd89baba762-kube-api-access-bf2x6\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:30:17.865405 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.865351 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkh4\" (UniqueName: \"kubernetes.io/projected/f4b96af0-063c-4752-a290-12371f09d52a-kube-api-access-zkkh4\") pod \"network-metrics-daemon-87jmt\" (UID: \"f4b96af0-063c-4752-a290-12371f09d52a\") " pod="openshift-multus/network-metrics-daemon-87jmt" Oct 31 00:30:17.865549 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.865504 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.865670 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.865544 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfvq9\" (UniqueName: \"kubernetes.io/projected/97194d65-9d7e-47b2-8c55-2283301e8a4d-kube-api-access-qfvq9\") pod \"dns-default-xbdxj\" (UID: \"97194d65-9d7e-47b2-8c55-2283301e8a4d\") " pod="openshift-dns/dns-default-xbdxj" Oct 31 00:30:17.865856 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.865834 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgv9\" (UniqueName: \"kubernetes.io/projected/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-kube-api-access-jcgv9\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.867920 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.867094 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppsr\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-kube-api-access-vppsr\") pod \"prometheus-k8s-0\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:17.867920 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.867678 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7sf\" (UniqueName: \"kubernetes.io/projected/b22fa0b3-c20e-402b-81b5-898a838d42c9-kube-api-access-pq7sf\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.964027 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.963985 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysconfig\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964027 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964027 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-modprobe-d\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964044 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-systemd\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964059 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-log-socket\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964077 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f79572d1-5d21-4771-b613-85a421e550b3-crio-socket\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964105 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-etc-selinux\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964130 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/658d1734-2e40-4784-8c85-d57ce4c5c7a8-host-slash\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964136 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysconfig\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964152 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-netns\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964162 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-modprobe-d\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964170 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f79572d1-5d21-4771-b613-85a421e550b3-crio-socket\") pod \"insights-runtime-extractor-g62jt\" (UID: \"f79572d1-5d21-4771-b613-85a421e550b3\") " pod="openshift-insights/insights-runtime-extractor-g62jt" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964144 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-log-socket\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964187 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/658d1734-2e40-4784-8c85-d57ce4c5c7a8-host-slash\") pod \"iptables-alerter-9l4x8\" (UID: \"658d1734-2e40-4784-8c85-d57ce4c5c7a8\") " pod="openshift-network-operator/iptables-alerter-9l4x8" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964195 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-etc-selinux\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964212 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-netns\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964144 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-systemd\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964179 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb2daade-8f58-42e8-a79b-2a373ff186ea-host\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964223 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb2daade-8f58-42e8-a79b-2a373ff186ea-host\") pod \"node-ca-bmn4m\" (UID: \"fb2daade-8f58-42e8-a79b-2a373ff186ea\") " pod="openshift-image-registry/node-ca-bmn4m" Oct 31 00:30:17.964272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964270 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-multus-certs\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964297 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-conf-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964322 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-ovn\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964359 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-multus-certs\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964370 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-ovn\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964367 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-conf-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964398 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysctl-conf\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964415 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-run\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964433 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964458 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964478 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-registration-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964485 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-run\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964498 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-kubernetes\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964514 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964522 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964519 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-cni-bin\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964537 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-kubernetes\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964540 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-registration-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964557 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-system-cni-dir\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964579 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-host\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964605 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-system-cni-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964557 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysctl-conf\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964610 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-system-cni-dir\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964645 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-cni-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964659 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-host\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964557 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-cni-bin\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964681 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-kubelet-config\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964685 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-system-cni-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964694 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-cni-dir\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964699 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-cnibin\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964730 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964760 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b02c695-144a-4481-bc14-1cd89baba762-hosts-file\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964743 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-kubelet-config\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964790 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysctl-d\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964802 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b02c695-144a-4481-bc14-1cd89baba762-hosts-file\") pod \"node-resolver-m8d27\" (UID: \"0b02c695-144a-4481-bc14-1cd89baba762\") " pod="openshift-dns/node-resolver-m8d27" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964808 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-dbus\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964808 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964824 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-etc-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964845 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-cni-bin\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964762 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50ef0d54-5212-427c-8bab-04b4262c3e11-cnibin\") pod \"multus-additional-cni-plugins-jqkmw\" (UID: \"50ef0d54-5212-427c-8bab-04b4262c3e11\") " pod="openshift-multus/multus-additional-cni-plugins-jqkmw" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964862 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-etc-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964869 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-lib-modules\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964877 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31d4d8b4-ee1c-4703-8ac5-f739d72ab13d-dbus\") pod \"global-pull-secret-syncer-wj84l\" (UID: \"31d4d8b4-ee1c-4703-8ac5-f739d72ab13d\") " pod="kube-system/global-pull-secret-syncer-wj84l" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964894 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-cni-bin\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964902 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-etc-sysctl-d\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964895 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-run-ovn-kubernetes\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964930 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-run-ovn-kubernetes\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964935 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-root\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964958 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-root\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964960 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-systemd-units\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964977 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-systemd-units\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964980 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-lib-modules\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.964998 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-cni-netd\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965014 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-slash\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.964995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965030 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-cni-multus\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965044 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-kubelet\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965057 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-var-lib-kubelet\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965016 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-cni-netd\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965071 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-cnibin\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965097 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-sys\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965106 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-slash\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965108 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-cni-multus\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965122 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-socket-dir-parent\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965136 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-var-lib-kubelet\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965136 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-cnibin\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965158 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-sys\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965161 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33c29ec2-dcd1-4c23-99cc-66c6684db03a-var-lib-kubelet\") pod \"tuned-pptjg\" (UID: \"33c29ec2-dcd1-4c23-99cc-66c6684db03a\") " pod="openshift-cluster-node-tuning-operator/tuned-pptjg" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965168 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-multus-socket-dir-parent\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965172 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-kubelet\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965194 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-k8s-cni-cncf-io\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965203 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-kubelet\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965210 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-etc-kubernetes\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965229 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-wtmp\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965250 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-host-run-k8s-cni-cncf-io\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965252 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-var-lib-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965258 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-etc-kubernetes\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965275 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-socket-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965281 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-var-lib-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965290 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-device-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965308 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-run-netns\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965311 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-node-exporter-wtmp\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965324 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-socket-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965332 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-device-dir\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965344 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-node-log\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965350 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-host-run-netns\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965391 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-node-log\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965389 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-sys\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965419 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-systemd\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965433 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-hostroot\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965450 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965465 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-sys-fs\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965486 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-systemd\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965493 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b22fa0b3-c20e-402b-81b5-898a838d42c9-sys\") pod \"node-exporter-7jm8z\" (UID: \"b22fa0b3-c20e-402b-81b5-898a838d42c9\") " pod="openshift-monitoring/node-exporter-7jm8z" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965510 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db73288c-815d-49ed-9cdb-6799fb6ecc7b-hostroot\") pod \"multus-9zlgx\" (UID: \"db73288c-815d-49ed-9cdb-6799fb6ecc7b\") " pod="openshift-multus/multus-9zlgx" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965524 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522-sys-fs\") pod \"aws-ebs-csi-driver-node-xst6c\" (UID: \"fbda83c1-6c9e-45d7-9cc5-0ff7f2cc5522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xst6c" Oct 31 00:30:17.966558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:17.965511 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/912d5cf1-39a7-4fe3-8e72-d6b20bd67476-run-openvswitch\") pod \"ovnkube-node-swvwl\" (UID: \"912d5cf1-39a7-4fe3-8e72-d6b20bd67476\") " pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:18.047746 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:18.047729 5821 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:30:18.157339 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:18.157317 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:18.640579 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:18.640551 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xbdxj" Oct 31 00:30:18.855519 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:18.855441 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nsk8c" event={"ID":"698ab510-f0c2-4745-b410-b1697d1aabb8","Type":"ContainerStarted","Data":"2d104582c32eb604d2cb5ec11fb6f9df3044b6d0d2e3b26837d8f1094d703a3c"} Oct 31 00:30:19.838161 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:19.838111 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:30:20.531493 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:20.531449 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:30:20.751557 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:20.751531 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:22.898854 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:22.898821 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rz5vm" Oct 31 00:30:23.615654 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:23.615614 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:23.615858 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:23.615846 5821 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:30:25.110029 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:25.109998 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-swvwl" Oct 31 00:30:29.929380 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:29.929300 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d4jrp" Oct 31 00:30:35.156802 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.156768 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:30:35.157246 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.157197 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="prometheus" containerID="cri-o://3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379" gracePeriod=600 Oct 31 00:30:35.157314 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.157239 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="kube-rbac-proxy" containerID="cri-o://2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5" gracePeriod=600 Oct 31 00:30:35.157484 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.157422 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="thanos-sidecar" containerID="cri-o://e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3" gracePeriod=600 Oct 31 00:30:35.158313 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.157814 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="config-reloader" containerID="cri-o://a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8" gracePeriod=600 Oct 31 00:30:35.158313 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.157826 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4" gracePeriod=600 Oct 31 00:30:35.158313 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.157857 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="kube-rbac-proxy-web" containerID="cri-o://620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2" gracePeriod=600 Oct 31 00:30:35.398858 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.398833 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:35.497140 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497109 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-thanos-prometheus-http-client-file\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.497299 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497145 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-0\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.497299 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497175 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-kube-rbac-proxy\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.497299 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497205 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-grpc-tls\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.497299 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497282 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.497484 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497321 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-serving-certs-ca-bundle\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.497484 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497355 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.497484 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497388 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497728 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-web-config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-web-config\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497772 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-tls\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497808 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-trusted-ca-bundle\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497853 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-2\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497901 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-metrics-client-certs\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497884 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-serving-certs-ca-bundle" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.497981 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-kubelet-serving-ca-bundle\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.498026 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-tls-assets\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-tls-assets\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.498057 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-kube-api-access-vppsr\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-kube-api-access-vppsr\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.498143 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config-out\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.498236 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-db\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.498322 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-metrics-client-ca\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.498354 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-1\") pod \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\" (UID: \"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2\") " Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.498468 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-2" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.498605 5821 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.498976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.498648 5821 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.499946 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.499915 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-trusted-ca-bundle" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:35.500035 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.499991 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-db" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:35.500715 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.500687 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-1" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:35.501032 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.501000 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-0" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:35.501151 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.501131 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-kube-rbac-proxy" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:35.501512 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.501456 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-grpc-tls" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:35.501633 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.501583 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:35.501828 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.501780 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-thanos-prometheus-http-client-file" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:35.502097 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.501943 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-metrics-client-ca" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:35.502171 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.502136 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-kubelet-serving-ca-bundle" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:35.502419 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.502393 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-tls" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:35.502757 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.502712 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-kube-rbac-proxy-web" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:35.502984 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.502959 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-kube-api-access-vppsr" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "kube-api-access-vppsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:35.502984 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.502966 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-thanos-sidecar-tls" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:35.503329 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.503301 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config-out" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:35.503818 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.503799 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-metrics-client-certs" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:35.503992 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.503975 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-tls-assets" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:35.511723 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.511696 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-web-config" pod "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" (UID: "1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:35.599115 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599092 5821 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599115 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599113 5821 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599123 5821 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-web-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599132 5821 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599143 5821 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599152 5821 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-metrics-client-certs\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599161 5821 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599170 5821 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-tls-assets\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599180 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vppsr\" (UniqueName: \"kubernetes.io/projected/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-kube-api-access-vppsr\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599188 5821 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-config-out\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599196 5821 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-db\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599205 5821 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-configmap-metrics-client-ca\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599213 5821 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599222 5821 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599230 5821 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599237 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599238 5821 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-kube-rbac-proxy\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599677 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599246 5821 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-grpc-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.599677 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.599254 5821 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.924504 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924471 5821 generic.go:356] "Generic (PLEG): container finished" podID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerID="9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4" exitCode=0 Oct 31 00:30:35.924504 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924497 5821 generic.go:356] "Generic (PLEG): container finished" podID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerID="2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5" exitCode=0 Oct 31 00:30:35.924504 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924503 5821 generic.go:356] "Generic (PLEG): container finished" podID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerID="620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2" exitCode=0 Oct 31 00:30:35.924504 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924509 5821 generic.go:356] "Generic (PLEG): container finished" podID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerID="e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3" exitCode=0 Oct 31 00:30:35.924868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924515 5821 generic.go:356] "Generic (PLEG): container finished" podID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerID="a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8" exitCode=0 Oct 31 00:30:35.924868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924521 5821 generic.go:356] "Generic (PLEG): container finished" podID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerID="3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379" exitCode=0 Oct 31 00:30:35.924868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924553 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerDied","Data":"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4"} Oct 31 00:30:35.924868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924611 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerDied","Data":"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5"} Oct 31 00:30:35.924868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924654 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:35.924868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924683 5821 scope.go:122] "RemoveContainer" containerID="9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4" Oct 31 00:30:35.924868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924661 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerDied","Data":"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2"} Oct 31 00:30:35.924868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924813 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerDied","Data":"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3"} Oct 31 00:30:35.924868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924870 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerDied","Data":"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8"} Oct 31 00:30:35.925285 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924915 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerDied","Data":"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379"} Oct 31 00:30:35.925285 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.924948 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2","Type":"ContainerDied","Data":"fcbd30c9a52d9925e349b62c93ae7b8f7e6377aeca2b79dc7f6bea32cc9a8918"} Oct 31 00:30:35.932297 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.932277 5821 scope.go:122] "RemoveContainer" containerID="2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5" Oct 31 00:30:35.939210 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.939193 5821 scope.go:122] "RemoveContainer" containerID="620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2" Oct 31 00:30:35.945805 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.945789 5821 scope.go:122] "RemoveContainer" containerID="e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3" Oct 31 00:30:35.948233 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.948213 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:30:35.953135 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.953117 5821 scope.go:122] "RemoveContainer" containerID="a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8" Oct 31 00:30:35.953258 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.953240 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:30:35.959682 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.959666 5821 scope.go:122] "RemoveContainer" containerID="3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379" Oct 31 00:30:35.966778 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.966761 5821 scope.go:122] "RemoveContainer" containerID="3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263" Oct 31 00:30:35.973351 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.973332 5821 scope.go:122] "RemoveContainer" containerID="9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4" Oct 31 00:30:35.973671 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:35.973612 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": container with ID starting with 9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4 not found: ID does not exist" containerID="9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4" Oct 31 00:30:35.973758 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.973685 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4"} err="failed to get container status \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": rpc error: code = NotFound desc = could not find container \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": container with ID starting with 9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4 not found: ID does not exist" Oct 31 00:30:35.973758 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.973727 5821 scope.go:122] "RemoveContainer" containerID="2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5" Oct 31 00:30:35.973981 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:35.973963 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": container with ID starting with 2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5 not found: ID does not exist" containerID="2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5" Oct 31 00:30:35.974041 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.973988 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5"} err="failed to get container status \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": rpc error: code = NotFound desc = could not find container \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": container with ID starting with 2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5 not found: ID does not exist" Oct 31 00:30:35.974041 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.974010 5821 scope.go:122] "RemoveContainer" containerID="620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2" Oct 31 00:30:35.974242 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:35.974228 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": container with ID starting with 620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2 not found: ID does not exist" containerID="620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2" Oct 31 00:30:35.974285 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.974246 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2"} err="failed to get container status \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": rpc error: code = NotFound desc = could not find container \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": container with ID starting with 620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2 not found: ID does not exist" Oct 31 00:30:35.974285 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.974259 5821 scope.go:122] "RemoveContainer" containerID="e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3" Oct 31 00:30:35.974469 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:35.974452 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": container with ID starting with e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3 not found: ID does not exist" containerID="e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3" Oct 31 00:30:35.974536 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.974477 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3"} err="failed to get container status \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": rpc error: code = NotFound desc = could not find container \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": container with ID starting with e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3 not found: ID does not exist" Oct 31 00:30:35.974536 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.974496 5821 scope.go:122] "RemoveContainer" containerID="a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8" Oct 31 00:30:35.974766 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:35.974731 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": container with ID starting with a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8 not found: ID does not exist" containerID="a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8" Oct 31 00:30:35.974818 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.974771 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8"} err="failed to get container status \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": rpc error: code = NotFound desc = could not find container \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": container with ID starting with a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8 not found: ID does not exist" Oct 31 00:30:35.974818 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.974783 5821 scope.go:122] "RemoveContainer" containerID="3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379" Oct 31 00:30:35.975047 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:35.975029 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": container with ID starting with 3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379 not found: ID does not exist" containerID="3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379" Oct 31 00:30:35.975094 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.975052 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379"} err="failed to get container status \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": rpc error: code = NotFound desc = could not find container \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": container with ID starting with 3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379 not found: ID does not exist" Oct 31 00:30:35.975094 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.975064 5821 scope.go:122] "RemoveContainer" containerID="3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263" Oct 31 00:30:35.975314 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:35.975298 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": container with ID starting with 3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263 not found: ID does not exist" containerID="3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263" Oct 31 00:30:35.975369 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.975323 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263"} err="failed to get container status \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": rpc error: code = NotFound desc = could not find container \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": container with ID starting with 3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263 not found: ID does not exist" Oct 31 00:30:35.975369 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.975342 5821 scope.go:122] "RemoveContainer" containerID="9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4" Oct 31 00:30:35.975582 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.975559 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4"} err="failed to get container status \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": rpc error: code = NotFound desc = could not find container \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": container with ID starting with 9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4 not found: ID does not exist" Oct 31 00:30:35.975582 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.975580 5821 scope.go:122] "RemoveContainer" containerID="2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5" Oct 31 00:30:35.975814 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.975795 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5"} err="failed to get container status \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": rpc error: code = NotFound desc = could not find container \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": container with ID starting with 2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5 not found: ID does not exist" Oct 31 00:30:35.975858 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.975814 5821 scope.go:122] "RemoveContainer" containerID="620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2" Oct 31 00:30:35.976058 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.976043 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2"} err="failed to get container status \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": rpc error: code = NotFound desc = could not find container \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": container with ID starting with 620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2 not found: ID does not exist" Oct 31 00:30:35.976103 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.976058 5821 scope.go:122] "RemoveContainer" containerID="e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3" Oct 31 00:30:35.976292 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.976273 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3"} err="failed to get container status \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": rpc error: code = NotFound desc = could not find container \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": container with ID starting with e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3 not found: ID does not exist" Oct 31 00:30:35.976366 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.976292 5821 scope.go:122] "RemoveContainer" containerID="a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8" Oct 31 00:30:35.976525 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.976509 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8"} err="failed to get container status \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": rpc error: code = NotFound desc = could not find container \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": container with ID starting with a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8 not found: ID does not exist" Oct 31 00:30:35.976582 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.976526 5821 scope.go:122] "RemoveContainer" containerID="3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379" Oct 31 00:30:35.976798 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.976779 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379"} err="failed to get container status \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": rpc error: code = NotFound desc = could not find container \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": container with ID starting with 3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379 not found: ID does not exist" Oct 31 00:30:35.976850 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.976801 5821 scope.go:122] "RemoveContainer" containerID="3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263" Oct 31 00:30:35.977016 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.976996 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263"} err="failed to get container status \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": rpc error: code = NotFound desc = could not find container \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": container with ID starting with 3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263 not found: ID does not exist" Oct 31 00:30:35.977078 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.977017 5821 scope.go:122] "RemoveContainer" containerID="9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4" Oct 31 00:30:35.977235 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.977220 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4"} err="failed to get container status \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": rpc error: code = NotFound desc = could not find container \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": container with ID starting with 9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4 not found: ID does not exist" Oct 31 00:30:35.977282 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.977236 5821 scope.go:122] "RemoveContainer" containerID="2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5" Oct 31 00:30:35.977484 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.977467 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5"} err="failed to get container status \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": rpc error: code = NotFound desc = could not find container \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": container with ID starting with 2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5 not found: ID does not exist" Oct 31 00:30:35.977484 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.977483 5821 scope.go:122] "RemoveContainer" containerID="620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2" Oct 31 00:30:35.977750 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.977732 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2"} err="failed to get container status \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": rpc error: code = NotFound desc = could not find container \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": container with ID starting with 620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2 not found: ID does not exist" Oct 31 00:30:35.977827 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.977751 5821 scope.go:122] "RemoveContainer" containerID="e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3" Oct 31 00:30:35.978001 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.977983 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3"} err="failed to get container status \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": rpc error: code = NotFound desc = could not find container \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": container with ID starting with e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3 not found: ID does not exist" Oct 31 00:30:35.978063 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.978000 5821 scope.go:122] "RemoveContainer" containerID="a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8" Oct 31 00:30:35.978230 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.978210 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8"} err="failed to get container status \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": rpc error: code = NotFound desc = could not find container \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": container with ID starting with a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8 not found: ID does not exist" Oct 31 00:30:35.978230 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.978230 5821 scope.go:122] "RemoveContainer" containerID="3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379" Oct 31 00:30:35.978434 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.978415 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379"} err="failed to get container status \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": rpc error: code = NotFound desc = could not find container \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": container with ID starting with 3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379 not found: ID does not exist" Oct 31 00:30:35.978434 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.978434 5821 scope.go:122] "RemoveContainer" containerID="3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263" Oct 31 00:30:35.978671 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.978655 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263"} err="failed to get container status \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": rpc error: code = NotFound desc = could not find container \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": container with ID starting with 3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263 not found: ID does not exist" Oct 31 00:30:35.978727 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.978672 5821 scope.go:122] "RemoveContainer" containerID="9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4" Oct 31 00:30:35.978874 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.978856 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4"} err="failed to get container status \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": rpc error: code = NotFound desc = could not find container \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": container with ID starting with 9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4 not found: ID does not exist" Oct 31 00:30:35.978916 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.978874 5821 scope.go:122] "RemoveContainer" containerID="2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5" Oct 31 00:30:35.979068 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.979052 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5"} err="failed to get container status \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": rpc error: code = NotFound desc = could not find container \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": container with ID starting with 2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5 not found: ID does not exist" Oct 31 00:30:35.979105 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.979069 5821 scope.go:122] "RemoveContainer" containerID="620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2" Oct 31 00:30:35.979255 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.979235 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2"} err="failed to get container status \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": rpc error: code = NotFound desc = could not find container \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": container with ID starting with 620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2 not found: ID does not exist" Oct 31 00:30:35.979295 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.979255 5821 scope.go:122] "RemoveContainer" containerID="e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3" Oct 31 00:30:35.979429 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.979412 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3"} err="failed to get container status \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": rpc error: code = NotFound desc = could not find container \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": container with ID starting with e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3 not found: ID does not exist" Oct 31 00:30:35.979468 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.979430 5821 scope.go:122] "RemoveContainer" containerID="a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8" Oct 31 00:30:35.979605 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.979591 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8"} err="failed to get container status \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": rpc error: code = NotFound desc = could not find container \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": container with ID starting with a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8 not found: ID does not exist" Oct 31 00:30:35.979671 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.979604 5821 scope.go:122] "RemoveContainer" containerID="3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379" Oct 31 00:30:35.979811 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.979795 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379"} err="failed to get container status \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": rpc error: code = NotFound desc = could not find container \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": container with ID starting with 3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379 not found: ID does not exist" Oct 31 00:30:35.979847 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.979811 5821 scope.go:122] "RemoveContainer" containerID="3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263" Oct 31 00:30:35.979985 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.979970 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263"} err="failed to get container status \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": rpc error: code = NotFound desc = could not find container \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": container with ID starting with 3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263 not found: ID does not exist" Oct 31 00:30:35.979985 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.979985 5821 scope.go:122] "RemoveContainer" containerID="9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4" Oct 31 00:30:35.980162 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.980143 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4"} err="failed to get container status \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": rpc error: code = NotFound desc = could not find container \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": container with ID starting with 9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4 not found: ID does not exist" Oct 31 00:30:35.980202 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.980162 5821 scope.go:122] "RemoveContainer" containerID="2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5" Oct 31 00:30:35.980354 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.980340 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5"} err="failed to get container status \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": rpc error: code = NotFound desc = could not find container \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": container with ID starting with 2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5 not found: ID does not exist" Oct 31 00:30:35.980397 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.980354 5821 scope.go:122] "RemoveContainer" containerID="620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2" Oct 31 00:30:35.980554 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.980540 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2"} err="failed to get container status \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": rpc error: code = NotFound desc = could not find container \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": container with ID starting with 620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2 not found: ID does not exist" Oct 31 00:30:35.980591 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.980554 5821 scope.go:122] "RemoveContainer" containerID="e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3" Oct 31 00:30:35.980805 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.980790 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3"} err="failed to get container status \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": rpc error: code = NotFound desc = could not find container \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": container with ID starting with e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3 not found: ID does not exist" Oct 31 00:30:35.980851 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.980805 5821 scope.go:122] "RemoveContainer" containerID="a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8" Oct 31 00:30:35.981024 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.981009 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8"} err="failed to get container status \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": rpc error: code = NotFound desc = could not find container \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": container with ID starting with a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8 not found: ID does not exist" Oct 31 00:30:35.981065 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.981024 5821 scope.go:122] "RemoveContainer" containerID="3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379" Oct 31 00:30:35.981223 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.981207 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379"} err="failed to get container status \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": rpc error: code = NotFound desc = could not find container \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": container with ID starting with 3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379 not found: ID does not exist" Oct 31 00:30:35.981259 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.981224 5821 scope.go:122] "RemoveContainer" containerID="3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263" Oct 31 00:30:35.981411 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.981396 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263"} err="failed to get container status \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": rpc error: code = NotFound desc = could not find container \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": container with ID starting with 3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263 not found: ID does not exist" Oct 31 00:30:35.981452 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.981412 5821 scope.go:122] "RemoveContainer" containerID="9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4" Oct 31 00:30:35.981661 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.981642 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4"} err="failed to get container status \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": rpc error: code = NotFound desc = could not find container \"9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4\": container with ID starting with 9a4c7032119baba2b79a82e4f4f9e4806ac7a5dfb1c7bfaae6046d93eb3a6cb4 not found: ID does not exist" Oct 31 00:30:35.981661 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.981660 5821 scope.go:122] "RemoveContainer" containerID="2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5" Oct 31 00:30:35.981982 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.981963 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5"} err="failed to get container status \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": rpc error: code = NotFound desc = could not find container \"2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5\": container with ID starting with 2eb3fdc9e886f2651ab61213a67299081bec8ddce9a13fc8e4b4d5ba7e918fe5 not found: ID does not exist" Oct 31 00:30:35.981982 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.981981 5821 scope.go:122] "RemoveContainer" containerID="620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2" Oct 31 00:30:35.982341 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982305 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2"} err="failed to get container status \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": rpc error: code = NotFound desc = could not find container \"620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2\": container with ID starting with 620fff1e953c9c1e5c9d8abcac51eb53b1c13aa329152e5001bf61e8cc0a60c2 not found: ID does not exist" Oct 31 00:30:35.982341 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982340 5821 scope.go:122] "RemoveContainer" containerID="e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982505 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982774 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3"} err="failed to get container status \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": rpc error: code = NotFound desc = could not find container \"e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3\": container with ID starting with e13b1038ee27a1de130665e04b78fc1b128c7dcb2832f8682ab570b6f8806fc3 not found: ID does not exist" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982801 5821 scope.go:122] "RemoveContainer" containerID="a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982848 5821 cpu_manager.go:414] "containerMap: removing container" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="kube-rbac-proxy-web" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982865 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="kube-rbac-proxy-web" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982877 5821 cpu_manager.go:414] "containerMap: removing container" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="thanos-sidecar" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982882 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="thanos-sidecar" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982896 5821 cpu_manager.go:414] "containerMap: removing container" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="prometheus" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982902 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="prometheus" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982915 5821 cpu_manager.go:414] "containerMap: removing container" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="kube-rbac-proxy-thanos" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982921 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="kube-rbac-proxy-thanos" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982927 5821 cpu_manager.go:414] "containerMap: removing container" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="kube-rbac-proxy" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982932 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="kube-rbac-proxy" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982946 5821 cpu_manager.go:414] "containerMap: removing container" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="config-reloader" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982951 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="config-reloader" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982956 5821 cpu_manager.go:414] "containerMap: removing container" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="init-config-reloader" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.982962 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="init-config-reloader" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.983007 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="kube-rbac-proxy-web" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.983015 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="thanos-sidecar" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.983021 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="prometheus" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.983026 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="kube-rbac-proxy-thanos" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.983031 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="kube-rbac-proxy" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.983038 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="config-reloader" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.983045 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" containerName="init-config-reloader" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.983073 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8"} err="failed to get container status \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": rpc error: code = NotFound desc = could not find container \"a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8\": container with ID starting with a221b0e927403516906617aa3f416911c32d3bef494a9cbfe2a5dfdcbb106dd8 not found: ID does not exist" Oct 31 00:30:35.983123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.983088 5821 scope.go:122] "RemoveContainer" containerID="3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379" Oct 31 00:30:35.984014 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.983323 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379"} err="failed to get container status \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": rpc error: code = NotFound desc = could not find container \"3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379\": container with ID starting with 3d488bad7dc3408e280418d19eb4c598149f31557a81d6f6f1d0c13b19588379 not found: ID does not exist" Oct 31 00:30:35.984014 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.983338 5821 scope.go:122] "RemoveContainer" containerID="3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263" Oct 31 00:30:35.984014 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.983574 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263"} err="failed to get container status \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": rpc error: code = NotFound desc = could not find container \"3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263\": container with ID starting with 3645b321b56a6b760f91febb7d17a065e60a94cb9c7fd75980e2639540d72263 not found: ID does not exist" Oct 31 00:30:35.986444 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.986426 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:35.990827 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.990806 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:30:35.990924 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.990854 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:30:35.990924 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.990861 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fmft110ekd91k\"" Oct 31 00:30:35.990924 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.990866 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:30:35.990924 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.990872 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jgktj\"" Oct 31 00:30:35.990924 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.990891 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:30:35.990924 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.990862 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:30:35.990924 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.990906 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:30:35.990924 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.990919 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:30:35.990924 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.990925 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:30:35.991259 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.990964 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:30:35.991259 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.991070 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:30:35.991259 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.991129 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:30:35.991259 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.991173 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:30:35.991466 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.991443 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:30:35.993109 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.993075 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:30:35.993332 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:35.993236 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:30:36.005427 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.005407 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:30:36.103037 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103006 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103037 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103039 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103227 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103058 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103227 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103077 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjbrn\" (UniqueName: \"kubernetes.io/projected/b2a548c0-6c71-4ffa-9172-67bd7536a791-kube-api-access-rjbrn\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103227 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103208 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2a548c0-6c71-4ffa-9172-67bd7536a791-config-out\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103227 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103224 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103382 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103243 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103382 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103257 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103382 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103271 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103470 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103374 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103470 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103404 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103470 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103437 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103470 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103467 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103584 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103487 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-config\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103584 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103507 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103584 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103530 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103584 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103568 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2a548c0-6c71-4ffa-9172-67bd7536a791-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103732 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103605 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103732 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103646 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-web-config\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.103732 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.103666 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.204181 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204128 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.204181 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204156 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.204181 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204178 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.204589 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204197 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.204589 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204239 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-config\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.204589 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204264 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.204589 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204440 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.204589 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204487 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2a548c0-6c71-4ffa-9172-67bd7536a791-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.204589 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204512 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.204589 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204538 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-web-config\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.204589 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204562 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.204589 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204566 5821 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:30:36.205061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204604 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204665 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204692 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204719 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjbrn\" (UniqueName: \"kubernetes.io/projected/b2a548c0-6c71-4ffa-9172-67bd7536a791-kube-api-access-rjbrn\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204775 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2a548c0-6c71-4ffa-9172-67bd7536a791-config-out\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204807 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204838 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204859 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.204884 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205061 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.205044 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205737 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.205224 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205737 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.205507 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205865 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.205840 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.205987 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.205967 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.206769 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.206585 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.207460 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.207430 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2a548c0-6c71-4ffa-9172-67bd7536a791-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.208408 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.207708 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.208408 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.207976 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.208408 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.208255 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.208641 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.208434 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-web-config\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.208707 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.208664 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.209515 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.209488 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.209701 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.209676 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.209879 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.209857 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b2a548c0-6c71-4ffa-9172-67bd7536a791-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.210188 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.210171 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2a548c0-6c71-4ffa-9172-67bd7536a791-config-out\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.210647 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.210602 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.210647 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.210613 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-config\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.210771 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.210753 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b2a548c0-6c71-4ffa-9172-67bd7536a791-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.213541 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.213523 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjbrn\" (UniqueName: \"kubernetes.io/projected/b2a548c0-6c71-4ffa-9172-67bd7536a791-kube-api-access-rjbrn\") pod \"prometheus-k8s-0\" (UID: \"b2a548c0-6c71-4ffa-9172-67bd7536a791\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.304594 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.304571 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.584676 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.584569 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:30:36.587918 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:30:36.587889 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a548c0_6c71_4ffa_9172_67bd7536a791.slice/crio-21e6a41fb8bc6affa0e24911ca999eb47646119f573f3636f9927c754fb46d91 WatchSource:0}: Error finding container 21e6a41fb8bc6affa0e24911ca999eb47646119f573f3636f9927c754fb46d91: Status 404 returned error can't find the container with id 21e6a41fb8bc6affa0e24911ca999eb47646119f573f3636f9927c754fb46d91 Oct 31 00:30:36.777595 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.777560 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2" path="/var/lib/kubelet/pods/1a2e4815-e7d6-44e0-bf9f-62ed9eaee9d2/volumes" Oct 31 00:30:36.929610 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.929577 5821 generic.go:356] "Generic (PLEG): container finished" podID="b2a548c0-6c71-4ffa-9172-67bd7536a791" containerID="3b5be40b396ff95563e20dc7a708a9c1f3f239bc5c323f7b5b16b7a1f5a79024" exitCode=0 Oct 31 00:30:36.929805 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.929673 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2a548c0-6c71-4ffa-9172-67bd7536a791","Type":"ContainerDied","Data":"3b5be40b396ff95563e20dc7a708a9c1f3f239bc5c323f7b5b16b7a1f5a79024"} Oct 31 00:30:36.929805 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:36.929759 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2a548c0-6c71-4ffa-9172-67bd7536a791","Type":"ContainerStarted","Data":"21e6a41fb8bc6affa0e24911ca999eb47646119f573f3636f9927c754fb46d91"} Oct 31 00:30:37.936718 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:37.936675 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2a548c0-6c71-4ffa-9172-67bd7536a791","Type":"ContainerStarted","Data":"c4bfea3b414bfdd98d9f905b7d146cef71123354769480655356b7158a78bb24"} Oct 31 00:30:37.937164 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:37.936789 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2a548c0-6c71-4ffa-9172-67bd7536a791","Type":"ContainerStarted","Data":"0b6846cda77aff31f0a54cd8cf2611504ae90b408d85e038215f1f64066b09e0"} Oct 31 00:30:37.937164 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:37.936842 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2a548c0-6c71-4ffa-9172-67bd7536a791","Type":"ContainerStarted","Data":"f1eca34149dabec659512265388b293f1220f49b3d12944539e6aa3783486a80"} Oct 31 00:30:37.937164 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:37.936878 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2a548c0-6c71-4ffa-9172-67bd7536a791","Type":"ContainerStarted","Data":"ddb36e58f26f68ca1827d2886f29424946f9cdab4b4a58f9aa57a36d20271835"} Oct 31 00:30:37.937164 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:37.936916 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2a548c0-6c71-4ffa-9172-67bd7536a791","Type":"ContainerStarted","Data":"2a96c234bc8647263d3afaa5978aa71a5e5ccdbdac8bc7b054d8ac6ef9f9e82b"} Oct 31 00:30:37.937164 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:37.936949 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2a548c0-6c71-4ffa-9172-67bd7536a791","Type":"ContainerStarted","Data":"6cc78a2c126498a90ed6781330211f5257b9f0f6a0fcbf2469e424b0a0a87ec2"} Oct 31 00:30:37.966000 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:37.965936 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.950338341 podStartE2EDuration="2.965909466s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.571125ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:37.965152222 +0000 UTC m=+21.326734861" watchObservedRunningTime="2026-10-31 00:30:37.965909466 +0000 UTC m=+21.327492105" Oct 31 00:30:41.305167 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:41.305129 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:49.837981 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:30:49.837945 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:30:50.533926 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:30:50.533895 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nsk8c" Oct 31 00:31:16.508696 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:31:16.508658 5821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:31:16.524780 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:31:16.524756 5821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:31:17.077800 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:31:17.077727 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:31:19.838458 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:31:19.838417 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:31:49.837477 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:31:49.837440 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:32:19.837924 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:32:19.837883 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:32:49.837479 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:32:49.837438 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:33:19.837647 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:33:19.837591 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:33:49.839189 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:33:49.839102 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:34:19.838284 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:34:19.838246 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:34:49.838152 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:34:49.838117 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:35:19.837847 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:35:19.837808 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:35:49.838205 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:35:49.838162 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:36:10.403124 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.403095 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-j7brj"] Oct 31 00:36:10.406104 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.406086 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-j7brj" Oct 31 00:36:10.409975 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.409950 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-j7brj"] Oct 31 00:36:10.425478 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.425459 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Oct 31 00:36:10.425611 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.425573 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mfqhl\"" Oct 31 00:36:10.451281 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.451262 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Oct 31 00:36:10.451381 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.451352 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Oct 31 00:36:10.527319 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.527293 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9547691d-0f50-476f-a4c1-8f084638d517-data\") pod \"seaweedfs-78bb9c6ff8-j7brj\" (UID: \"9547691d-0f50-476f-a4c1-8f084638d517\") " pod="kserve/seaweedfs-78bb9c6ff8-j7brj" Oct 31 00:36:10.527435 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.527326 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bffnl\" (UniqueName: \"kubernetes.io/projected/9547691d-0f50-476f-a4c1-8f084638d517-kube-api-access-bffnl\") pod \"seaweedfs-78bb9c6ff8-j7brj\" (UID: \"9547691d-0f50-476f-a4c1-8f084638d517\") " pod="kserve/seaweedfs-78bb9c6ff8-j7brj" Oct 31 00:36:10.628332 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.628300 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9547691d-0f50-476f-a4c1-8f084638d517-data\") pod \"seaweedfs-78bb9c6ff8-j7brj\" (UID: \"9547691d-0f50-476f-a4c1-8f084638d517\") " pod="kserve/seaweedfs-78bb9c6ff8-j7brj" Oct 31 00:36:10.628471 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.628340 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffnl\" (UniqueName: \"kubernetes.io/projected/9547691d-0f50-476f-a4c1-8f084638d517-kube-api-access-bffnl\") pod \"seaweedfs-78bb9c6ff8-j7brj\" (UID: \"9547691d-0f50-476f-a4c1-8f084638d517\") " pod="kserve/seaweedfs-78bb9c6ff8-j7brj" Oct 31 00:36:10.628697 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.628678 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9547691d-0f50-476f-a4c1-8f084638d517-data\") pod \"seaweedfs-78bb9c6ff8-j7brj\" (UID: \"9547691d-0f50-476f-a4c1-8f084638d517\") " pod="kserve/seaweedfs-78bb9c6ff8-j7brj" Oct 31 00:36:10.636878 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.636857 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bffnl\" (UniqueName: \"kubernetes.io/projected/9547691d-0f50-476f-a4c1-8f084638d517-kube-api-access-bffnl\") pod \"seaweedfs-78bb9c6ff8-j7brj\" (UID: \"9547691d-0f50-476f-a4c1-8f084638d517\") " pod="kserve/seaweedfs-78bb9c6ff8-j7brj" Oct 31 00:36:10.718247 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.718215 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-j7brj" Oct 31 00:36:10.984705 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.984603 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-j7brj"] Oct 31 00:36:10.988532 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:36:10.988508 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9547691d_0f50_476f_a4c1_8f084638d517.slice/crio-9ef535b3ad91ded97758fab23bebcb96f7c2792b23b1da5b410b4b85659c6ecf WatchSource:0}: Error finding container 9ef535b3ad91ded97758fab23bebcb96f7c2792b23b1da5b410b4b85659c6ecf: Status 404 returned error can't find the container with id 9ef535b3ad91ded97758fab23bebcb96f7c2792b23b1da5b410b4b85659c6ecf Oct 31 00:36:10.990101 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:10.990086 5821 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:36:11.993951 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:11.993912 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-j7brj" event={"ID":"9547691d-0f50-476f-a4c1-8f084638d517","Type":"ContainerStarted","Data":"9ef535b3ad91ded97758fab23bebcb96f7c2792b23b1da5b410b4b85659c6ecf"} Oct 31 00:36:16.745614 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:16.745583 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Oct 31 00:36:17.010515 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:17.010432 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-j7brj" event={"ID":"9547691d-0f50-476f-a4c1-8f084638d517","Type":"ContainerStarted","Data":"eeb9dcac7dcabe6c66c073116364ca4e860c313dddf34c966f95eb68353d643b"} Oct 31 00:36:17.010672 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:17.010555 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-78bb9c6ff8-j7brj" Oct 31 00:36:17.036323 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:17.036282 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-78bb9c6ff8-j7brj" podStartSLOduration=1.283547886 podStartE2EDuration="7.03627165s" totalImagesPullingTime="5.752723764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:17.036082871 +0000 UTC m=+360.397665510" watchObservedRunningTime="2026-10-31 00:36:17.03627165 +0000 UTC m=+360.397854285" Oct 31 00:36:19.837721 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:36:19.837686 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:36:23.015675 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:36:23.015642 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-78bb9c6ff8-j7brj" Oct 31 00:36:49.837801 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:36:49.837716 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:37:19.838413 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:37:19.838375 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:37:33.483796 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.483754 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-flp4s"] Oct 31 00:37:33.486883 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.486859 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:33.497527 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.497503 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-flp4s"] Oct 31 00:37:33.557038 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.557011 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Oct 31 00:37:33.557213 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.557154 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\"" Oct 31 00:37:33.557311 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.557294 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-cplqw\"" Oct 31 00:37:33.641115 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.641090 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a57c127-9e8c-482a-9511-d3959122f5be-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-flp4s\" (UID: \"7a57c127-9e8c-482a-9511-d3959122f5be\") " pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:33.641289 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.641126 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a57c127-9e8c-482a-9511-d3959122f5be-cert\") pod \"odh-model-controller-7f9f8cdc65-flp4s\" (UID: \"7a57c127-9e8c-482a-9511-d3959122f5be\") " pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:33.641289 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.641230 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbtv\" (UniqueName: \"kubernetes.io/projected/7a57c127-9e8c-482a-9511-d3959122f5be-kube-api-access-tsbtv\") pod \"odh-model-controller-7f9f8cdc65-flp4s\" (UID: \"7a57c127-9e8c-482a-9511-d3959122f5be\") " pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:33.742400 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.742319 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a57c127-9e8c-482a-9511-d3959122f5be-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-flp4s\" (UID: \"7a57c127-9e8c-482a-9511-d3959122f5be\") " pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:33.742400 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.742354 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a57c127-9e8c-482a-9511-d3959122f5be-cert\") pod \"odh-model-controller-7f9f8cdc65-flp4s\" (UID: \"7a57c127-9e8c-482a-9511-d3959122f5be\") " pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:33.742400 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.742376 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbtv\" (UniqueName: \"kubernetes.io/projected/7a57c127-9e8c-482a-9511-d3959122f5be-kube-api-access-tsbtv\") pod \"odh-model-controller-7f9f8cdc65-flp4s\" (UID: \"7a57c127-9e8c-482a-9511-d3959122f5be\") " pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:33.744897 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.744861 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a57c127-9e8c-482a-9511-d3959122f5be-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-flp4s\" (UID: \"7a57c127-9e8c-482a-9511-d3959122f5be\") " pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:33.744897 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.744885 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a57c127-9e8c-482a-9511-d3959122f5be-cert\") pod \"odh-model-controller-7f9f8cdc65-flp4s\" (UID: \"7a57c127-9e8c-482a-9511-d3959122f5be\") " pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:33.751038 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.751018 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbtv\" (UniqueName: \"kubernetes.io/projected/7a57c127-9e8c-482a-9511-d3959122f5be-kube-api-access-tsbtv\") pod \"odh-model-controller-7f9f8cdc65-flp4s\" (UID: \"7a57c127-9e8c-482a-9511-d3959122f5be\") " pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:33.798885 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:33.798862 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:34.067271 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:34.067205 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-flp4s"] Oct 31 00:37:34.070971 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:37:34.070942 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a57c127_9e8c_482a_9511_d3959122f5be.slice/crio-6c49a1f661dba9a60e0f9ecb164acc0ae13924415b30e7a0cc19e5195181785d WatchSource:0}: Error finding container 6c49a1f661dba9a60e0f9ecb164acc0ae13924415b30e7a0cc19e5195181785d: Status 404 returned error can't find the container with id 6c49a1f661dba9a60e0f9ecb164acc0ae13924415b30e7a0cc19e5195181785d Oct 31 00:37:34.260104 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:34.260066 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" event={"ID":"7a57c127-9e8c-482a-9511-d3959122f5be","Type":"ContainerStarted","Data":"6c49a1f661dba9a60e0f9ecb164acc0ae13924415b30e7a0cc19e5195181785d"} Oct 31 00:37:38.277994 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:38.277957 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" event={"ID":"7a57c127-9e8c-482a-9511-d3959122f5be","Type":"ContainerStarted","Data":"778b69d50d2bcd4107d3c43a6fdb8ff40ec18a03e1bcefee000013c502b92344"} Oct 31 00:37:38.278427 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:38.278120 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:38.306195 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:38.306144 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" podStartSLOduration=2.062259383 podStartE2EDuration="5.306121772s" totalImagesPullingTime="3.243862389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:38.305648094 +0000 UTC m=+441.667230732" watchObservedRunningTime="2026-10-31 00:37:38.306121772 +0000 UTC m=+441.667704407" Oct 31 00:37:49.283875 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:49.283844 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7f9f8cdc65-flp4s" Oct 31 00:37:49.837361 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:37:49.837320 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:37:50.656688 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:50.656658 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-init-9tj6m"] Oct 31 00:37:50.660247 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:50.660227 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9tj6m" Oct 31 00:37:50.663650 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:50.663607 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Oct 31 00:37:50.667727 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:50.667703 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9tj6m"] Oct 31 00:37:50.675008 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:50.674986 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zpwr\" (UniqueName: \"kubernetes.io/projected/b6dcf8ed-2f27-4841-9d44-985a060a4710-kube-api-access-4zpwr\") pod \"s3-init-9tj6m\" (UID: \"b6dcf8ed-2f27-4841-9d44-985a060a4710\") " pod="kserve/s3-init-9tj6m" Oct 31 00:37:50.675136 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:50.675110 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6dcf8ed-2f27-4841-9d44-985a060a4710-model-cache\") pod \"s3-init-9tj6m\" (UID: \"b6dcf8ed-2f27-4841-9d44-985a060a4710\") " pod="kserve/s3-init-9tj6m" Oct 31 00:37:50.775526 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:50.775499 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpwr\" (UniqueName: \"kubernetes.io/projected/b6dcf8ed-2f27-4841-9d44-985a060a4710-kube-api-access-4zpwr\") pod \"s3-init-9tj6m\" (UID: \"b6dcf8ed-2f27-4841-9d44-985a060a4710\") " pod="kserve/s3-init-9tj6m" Oct 31 00:37:50.775689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:50.775540 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6dcf8ed-2f27-4841-9d44-985a060a4710-model-cache\") pod \"s3-init-9tj6m\" (UID: \"b6dcf8ed-2f27-4841-9d44-985a060a4710\") " pod="kserve/s3-init-9tj6m" Oct 31 00:37:50.775991 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:50.775969 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6dcf8ed-2f27-4841-9d44-985a060a4710-model-cache\") pod \"s3-init-9tj6m\" (UID: \"b6dcf8ed-2f27-4841-9d44-985a060a4710\") " pod="kserve/s3-init-9tj6m" Oct 31 00:37:50.785069 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:50.785049 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zpwr\" (UniqueName: \"kubernetes.io/projected/b6dcf8ed-2f27-4841-9d44-985a060a4710-kube-api-access-4zpwr\") pod \"s3-init-9tj6m\" (UID: \"b6dcf8ed-2f27-4841-9d44-985a060a4710\") " pod="kserve/s3-init-9tj6m" Oct 31 00:37:50.982715 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:50.982686 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9tj6m" Oct 31 00:37:51.250999 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:51.250922 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9tj6m"] Oct 31 00:37:51.254401 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:37:51.254377 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6dcf8ed_2f27_4841_9d44_985a060a4710.slice/crio-565fded0e6d7df72b0940fd6320cefa53bafb1c7e0176e2c842dff6285f7f5ad WatchSource:0}: Error finding container 565fded0e6d7df72b0940fd6320cefa53bafb1c7e0176e2c842dff6285f7f5ad: Status 404 returned error can't find the container with id 565fded0e6d7df72b0940fd6320cefa53bafb1c7e0176e2c842dff6285f7f5ad Oct 31 00:37:51.324962 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:51.324925 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9tj6m" event={"ID":"b6dcf8ed-2f27-4841-9d44-985a060a4710","Type":"ContainerStarted","Data":"565fded0e6d7df72b0940fd6320cefa53bafb1c7e0176e2c842dff6285f7f5ad"} Oct 31 00:37:59.355439 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:37:59.355404 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9tj6m" event={"ID":"b6dcf8ed-2f27-4841-9d44-985a060a4710","Type":"ContainerStarted","Data":"9fba33dea7b51c181d292ab41f2c959c03d3e3496d5be9be2a1aea9836a3674a"} Oct 31 00:38:01.364675 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:38:01.364642 5821 generic.go:356] "Generic (PLEG): container finished" podID="b6dcf8ed-2f27-4841-9d44-985a060a4710" containerID="9fba33dea7b51c181d292ab41f2c959c03d3e3496d5be9be2a1aea9836a3674a" exitCode=0 Oct 31 00:38:01.364675 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:38:01.364651 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9tj6m" event={"ID":"b6dcf8ed-2f27-4841-9d44-985a060a4710","Type":"ContainerDied","Data":"9fba33dea7b51c181d292ab41f2c959c03d3e3496d5be9be2a1aea9836a3674a"} Oct 31 00:38:06.386056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:38:06.386018 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9tj6m" event={"ID":"b6dcf8ed-2f27-4841-9d44-985a060a4710","Type":"ContainerStarted","Data":"2267feb6fb724c403cb36eccc27c386e8e8d1e5f23ddc981a6a7ebb8a8d0754a"} Oct 31 00:38:06.403254 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:38:06.403195 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-init-9tj6m" podStartSLOduration=2.997489275 podStartE2EDuration="16.403179134s" totalImagesPullingTime="11.996538994s" totalInitContainerRuntime="1.409150865s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:06.401687411 +0000 UTC m=+469.763270054" watchObservedRunningTime="2026-10-31 00:38:06.403179134 +0000 UTC m=+469.764761775" Oct 31 00:38:19.838370 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:38:19.838321 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:38:49.838379 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:38:49.838339 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:39:11.605956 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:11.605919 5821 generic.go:356] "Generic (PLEG): container finished" podID="b6dcf8ed-2f27-4841-9d44-985a060a4710" containerID="2267feb6fb724c403cb36eccc27c386e8e8d1e5f23ddc981a6a7ebb8a8d0754a" exitCode=0 Oct 31 00:39:11.606393 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:11.605966 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9tj6m" event={"ID":"b6dcf8ed-2f27-4841-9d44-985a060a4710","Type":"ContainerDied","Data":"2267feb6fb724c403cb36eccc27c386e8e8d1e5f23ddc981a6a7ebb8a8d0754a"} Oct 31 00:39:12.731279 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:12.731258 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9tj6m" Oct 31 00:39:12.802282 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:12.802257 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6dcf8ed-2f27-4841-9d44-985a060a4710-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6dcf8ed-2f27-4841-9d44-985a060a4710-model-cache\") pod \"b6dcf8ed-2f27-4841-9d44-985a060a4710\" (UID: \"b6dcf8ed-2f27-4841-9d44-985a060a4710\") " Oct 31 00:39:12.802405 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:12.802313 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6dcf8ed-2f27-4841-9d44-985a060a4710-kube-api-access-4zpwr\" (UniqueName: \"kubernetes.io/projected/b6dcf8ed-2f27-4841-9d44-985a060a4710-kube-api-access-4zpwr\") pod \"b6dcf8ed-2f27-4841-9d44-985a060a4710\" (UID: \"b6dcf8ed-2f27-4841-9d44-985a060a4710\") " Oct 31 00:39:12.804529 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:12.804503 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6dcf8ed-2f27-4841-9d44-985a060a4710-kube-api-access-4zpwr" pod "b6dcf8ed-2f27-4841-9d44-985a060a4710" (UID: "b6dcf8ed-2f27-4841-9d44-985a060a4710"). InnerVolumeSpecName "kube-api-access-4zpwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:12.820427 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:12.820405 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6dcf8ed-2f27-4841-9d44-985a060a4710-model-cache" pod "b6dcf8ed-2f27-4841-9d44-985a060a4710" (UID: "b6dcf8ed-2f27-4841-9d44-985a060a4710"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:39:12.903535 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:12.903482 5821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6dcf8ed-2f27-4841-9d44-985a060a4710-model-cache\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:12.903535 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:12.903506 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zpwr\" (UniqueName: \"kubernetes.io/projected/b6dcf8ed-2f27-4841-9d44-985a060a4710-kube-api-access-4zpwr\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:13.613654 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:13.613600 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9tj6m" event={"ID":"b6dcf8ed-2f27-4841-9d44-985a060a4710","Type":"ContainerDied","Data":"565fded0e6d7df72b0940fd6320cefa53bafb1c7e0176e2c842dff6285f7f5ad"} Oct 31 00:39:13.613654 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:13.613661 5821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="565fded0e6d7df72b0940fd6320cefa53bafb1c7e0176e2c842dff6285f7f5ad" Oct 31 00:39:13.613853 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:13.613610 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9tj6m" Oct 31 00:39:19.838004 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:39:19.837966 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:39:33.950342 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.950255 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr"] Oct 31 00:39:33.952792 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.950698 5821 cpu_manager.go:414] "containerMap: removing container" podUID="b6dcf8ed-2f27-4841-9d44-985a060a4710" containerName="s3-init" Oct 31 00:39:33.952792 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.950715 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6dcf8ed-2f27-4841-9d44-985a060a4710" containerName="s3-init" Oct 31 00:39:33.952792 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.950750 5821 cpu_manager.go:414] "containerMap: removing container" podUID="b6dcf8ed-2f27-4841-9d44-985a060a4710" containerName="download-hf-model" Oct 31 00:39:33.952792 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.950759 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6dcf8ed-2f27-4841-9d44-985a060a4710" containerName="download-hf-model" Oct 31 00:39:33.952792 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.950830 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6dcf8ed-2f27-4841-9d44-985a060a4710" containerName="s3-init" Oct 31 00:39:33.953745 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.953727 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:33.963708 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.963641 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr"] Oct 31 00:39:33.964135 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.964117 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"kube-root-ca.crt\"" Oct 31 00:39:33.964300 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.964279 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"seaweedfs-s3-creds\"" Oct 31 00:39:33.964300 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.964295 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"openshift-service-ca.crt\"" Oct 31 00:39:33.964300 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.964326 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Oct 31 00:39:33.964566 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.964409 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:39:33.964958 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.964939 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"default-dockercfg-84b4v\"" Oct 31 00:39:33.965043 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:33.964967 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"isvc-canary-rollback-v2-predictor-serving-cert\"" Oct 31 00:39:34.055861 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.055830 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0b8382-f79e-46f2-986c-4a1680707822-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.056001 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.055868 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.056001 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.055902 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f0b8382-f79e-46f2-986c-4a1680707822-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.056001 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.055979 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snpzj\" (UniqueName: \"kubernetes.io/projected/9f0b8382-f79e-46f2-986c-4a1680707822-kube-api-access-snpzj\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.056111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.056029 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.123377 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.123346 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk"] Oct 31 00:39:34.126700 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.126682 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.130203 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.130178 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"isvc-canary-rollback-predictor-serving-cert\"" Oct 31 00:39:34.135463 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.135441 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk"] Oct 31 00:39:34.156944 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.156911 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.157117 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.156968 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0b8382-f79e-46f2-986c-4a1680707822-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.157117 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.157001 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.157117 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.157039 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f0b8382-f79e-46f2-986c-4a1680707822-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.157117 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.157068 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snpzj\" (UniqueName: \"kubernetes.io/projected/9f0b8382-f79e-46f2-986c-4a1680707822-kube-api-access-snpzj\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.157640 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.157595 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.157773 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.157753 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f0b8382-f79e-46f2-986c-4a1680707822-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.158099 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.158068 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.159673 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.159655 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0b8382-f79e-46f2-986c-4a1680707822-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.173252 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.173228 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snpzj\" (UniqueName: \"kubernetes.io/projected/9f0b8382-f79e-46f2-986c-4a1680707822-kube-api-access-snpzj\") pod \"isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.254867 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.254780 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t"] Oct 31 00:39:34.258130 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.258106 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.258255 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.258151 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.258255 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.258176 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.258323 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.258288 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ce13928-1a73-4459-8a6b-8e37e8d2160e-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.258379 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.258361 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgxrv\" (UniqueName: \"kubernetes.io/projected/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-api-access-zgxrv\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.258520 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.258508 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.265524 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.265498 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t"] Oct 31 00:39:34.271116 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.271096 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:39:34.282877 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.282852 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"openshift-service-ca.crt\"" Oct 31 00:39:34.282961 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.282852 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:39:34.282961 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.282896 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"kube-root-ca.crt\"" Oct 31 00:39:34.282961 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.282907 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-raw-sklearn-batcher-757b4-kube-rbac-proxy-sar-config\"" Oct 31 00:39:34.330508 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.330482 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"seaweedfs-s3-creds\"" Oct 31 00:39:34.330508 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.330502 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"default-dockercfg-gs5fn\"" Oct 31 00:39:34.330729 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.330576 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-raw-sklearn-batcher-757b4-predictor-serving-cert\"" Oct 31 00:39:34.358973 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.358943 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/414b332d-83f8-4e88-97de-156170e1cba9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.359100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.358984 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.359100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.359001 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.359172 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.359106 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ce13928-1a73-4459-8a6b-8e37e8d2160e-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.359235 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.359216 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxrv\" (UniqueName: \"kubernetes.io/projected/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-api-access-zgxrv\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.359287 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.359270 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.359340 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.359308 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.359397 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.359342 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.359397 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.359351 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.359397 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.359368 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/414b332d-83f8-4e88-97de-156170e1cba9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.359537 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.359494 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9jpl\" (UniqueName: \"kubernetes.io/projected/414b332d-83f8-4e88-97de-156170e1cba9-kube-api-access-z9jpl\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.360017 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.359966 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.360017 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.359982 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.361699 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.361675 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ce13928-1a73-4459-8a6b-8e37e8d2160e-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.368317 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.368296 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgxrv\" (UniqueName: \"kubernetes.io/projected/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-api-access-zgxrv\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-bzszk\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.444639 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.444584 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:39:34.460659 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.460635 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/414b332d-83f8-4e88-97de-156170e1cba9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.460768 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.460677 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jpl\" (UniqueName: \"kubernetes.io/projected/414b332d-83f8-4e88-97de-156170e1cba9-kube-api-access-z9jpl\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.460768 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.460703 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/414b332d-83f8-4e88-97de-156170e1cba9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.460768 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.460754 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.460916 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.460775 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.461242 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.461217 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/414b332d-83f8-4e88-97de-156170e1cba9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.461439 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.461420 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.461772 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.461749 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.463198 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.463174 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/414b332d-83f8-4e88-97de-156170e1cba9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.468908 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.468887 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9jpl\" (UniqueName: \"kubernetes.io/projected/414b332d-83f8-4e88-97de-156170e1cba9-kube-api-access-z9jpl\") pod \"isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.543591 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.543520 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr"] Oct 31 00:39:34.546274 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:39:34.546245 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0b8382_f79e_46f2_986c_4a1680707822.slice/crio-1844d664f217d27e10e4636852e2fdce4c36770c21a69a438a425db96b16b098 WatchSource:0}: Error finding container 1844d664f217d27e10e4636852e2fdce4c36770c21a69a438a425db96b16b098: Status 404 returned error can't find the container with id 1844d664f217d27e10e4636852e2fdce4c36770c21a69a438a425db96b16b098 Oct 31 00:39:34.580393 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.580361 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:39:34.686319 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.686289 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" event={"ID":"9f0b8382-f79e-46f2-986c-4a1680707822","Type":"ContainerStarted","Data":"300b426e005f70d89a3cc6679a1bcd8511008288b953874db522fbfdd8a3eb51"} Oct 31 00:39:34.686461 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.686354 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" event={"ID":"9f0b8382-f79e-46f2-986c-4a1680707822","Type":"ContainerStarted","Data":"1844d664f217d27e10e4636852e2fdce4c36770c21a69a438a425db96b16b098"} Oct 31 00:39:34.715148 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.714838 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk"] Oct 31 00:39:34.719063 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:39:34.719036 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce13928_1a73_4459_8a6b_8e37e8d2160e.slice/crio-ddd957a278d41f769f95e60ad2a6f3457636c712910294f53b81cda9c88dc3ae WatchSource:0}: Error finding container ddd957a278d41f769f95e60ad2a6f3457636c712910294f53b81cda9c88dc3ae: Status 404 returned error can't find the container with id ddd957a278d41f769f95e60ad2a6f3457636c712910294f53b81cda9c88dc3ae Oct 31 00:39:34.856565 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:34.856496 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t"] Oct 31 00:39:34.860229 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:39:34.860202 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414b332d_83f8_4e88_97de_156170e1cba9.slice/crio-5d3ba1d95efc742b1f079e953acc18ad39e04cfb8e992f8627515b2b37156b5a WatchSource:0}: Error finding container 5d3ba1d95efc742b1f079e953acc18ad39e04cfb8e992f8627515b2b37156b5a: Status 404 returned error can't find the container with id 5d3ba1d95efc742b1f079e953acc18ad39e04cfb8e992f8627515b2b37156b5a Oct 31 00:39:35.691829 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:35.691795 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" event={"ID":"2ce13928-1a73-4459-8a6b-8e37e8d2160e","Type":"ContainerStarted","Data":"8f9ff64f0a03fe52388c98970f5dd45ad98694f3a21abfafbfdab67b8c562e48"} Oct 31 00:39:35.692250 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:35.691876 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" event={"ID":"2ce13928-1a73-4459-8a6b-8e37e8d2160e","Type":"ContainerStarted","Data":"ddd957a278d41f769f95e60ad2a6f3457636c712910294f53b81cda9c88dc3ae"} Oct 31 00:39:35.693469 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:35.693442 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" event={"ID":"414b332d-83f8-4e88-97de-156170e1cba9","Type":"ContainerStarted","Data":"3aa629e43f845af946b82defe14cdf9a2a106ecf15ddcb13e7a063db7ad562b3"} Oct 31 00:39:35.693599 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:35.693506 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" event={"ID":"414b332d-83f8-4e88-97de-156170e1cba9","Type":"ContainerStarted","Data":"5d3ba1d95efc742b1f079e953acc18ad39e04cfb8e992f8627515b2b37156b5a"} Oct 31 00:39:38.712152 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:38.712116 5821 generic.go:356] "Generic (PLEG): container finished" podID="9f0b8382-f79e-46f2-986c-4a1680707822" containerID="300b426e005f70d89a3cc6679a1bcd8511008288b953874db522fbfdd8a3eb51" exitCode=0 Oct 31 00:39:38.712533 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:38.712155 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" event={"ID":"9f0b8382-f79e-46f2-986c-4a1680707822","Type":"ContainerDied","Data":"300b426e005f70d89a3cc6679a1bcd8511008288b953874db522fbfdd8a3eb51"} Oct 31 00:39:39.723674 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:39.723591 5821 generic.go:356] "Generic (PLEG): container finished" podID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerID="8f9ff64f0a03fe52388c98970f5dd45ad98694f3a21abfafbfdab67b8c562e48" exitCode=0 Oct 31 00:39:39.724082 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:39.723710 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" event={"ID":"2ce13928-1a73-4459-8a6b-8e37e8d2160e","Type":"ContainerDied","Data":"8f9ff64f0a03fe52388c98970f5dd45ad98694f3a21abfafbfdab67b8c562e48"} Oct 31 00:39:39.732346 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:39.732292 5821 generic.go:356] "Generic (PLEG): container finished" podID="414b332d-83f8-4e88-97de-156170e1cba9" containerID="3aa629e43f845af946b82defe14cdf9a2a106ecf15ddcb13e7a063db7ad562b3" exitCode=0 Oct 31 00:39:39.732464 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:39:39.732395 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" event={"ID":"414b332d-83f8-4e88-97de-156170e1cba9","Type":"ContainerDied","Data":"3aa629e43f845af946b82defe14cdf9a2a106ecf15ddcb13e7a063db7ad562b3"} Oct 31 00:39:49.878439 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:39:49.878396 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:40:04.910799 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:04.910762 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" event={"ID":"414b332d-83f8-4e88-97de-156170e1cba9","Type":"ContainerStarted","Data":"8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570"} Oct 31 00:40:05.917534 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:05.917499 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" event={"ID":"2ce13928-1a73-4459-8a6b-8e37e8d2160e","Type":"ContainerStarted","Data":"c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc"} Oct 31 00:40:05.920246 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:05.920215 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" event={"ID":"9f0b8382-f79e-46f2-986c-4a1680707822","Type":"ContainerStarted","Data":"2eec6c1e0ac0719ab24fa169be71cb41ef090107d847a87f0e2b661bd2a417e4"} Oct 31 00:40:08.936485 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:08.936450 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" event={"ID":"9f0b8382-f79e-46f2-986c-4a1680707822","Type":"ContainerStarted","Data":"394b7b3ad9fc7aa121d3102f2f3361621cd6f05310c9286167f47f99ee843aef"} Oct 31 00:40:08.936972 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:08.936581 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:40:08.938588 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:08.938567 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" event={"ID":"414b332d-83f8-4e88-97de-156170e1cba9","Type":"ContainerStarted","Data":"d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308"} Oct 31 00:40:08.961515 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:08.961469 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podStartSLOduration=2.21181722 podStartE2EDuration="35.961456509s" totalImagesPullingTime="29.890175495s" totalInitContainerRuntime="3.859463794s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:08.960969556 +0000 UTC m=+592.322552195" watchObservedRunningTime="2026-10-31 00:40:08.961456509 +0000 UTC m=+592.323039148" Oct 31 00:40:09.944439 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:09.944404 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" event={"ID":"2ce13928-1a73-4459-8a6b-8e37e8d2160e","Type":"ContainerStarted","Data":"c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d"} Oct 31 00:40:09.944909 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:09.944673 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:40:09.944909 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:09.944728 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:40:09.944909 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:09.944762 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:40:09.945690 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:09.945662 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused" Oct 31 00:40:09.945765 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:09.945670 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused" Oct 31 00:40:09.963694 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:09.963611 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podStartSLOduration=2.778021425 podStartE2EDuration="35.96359397s" totalImagesPullingTime="29.16907007s" totalInitContainerRuntime="4.016502475s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:09.962765215 +0000 UTC m=+593.324347856" watchObservedRunningTime="2026-10-31 00:40:09.96359397 +0000 UTC m=+593.325176610" Oct 31 00:40:10.947967 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:10.947922 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused" Oct 31 00:40:10.948422 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:10.948061 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused" Oct 31 00:40:15.953645 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:15.953598 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:40:15.954303 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:15.953969 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:40:15.954303 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:15.954281 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused" Oct 31 00:40:15.954909 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:15.954881 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused" Oct 31 00:40:16.972557 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:16.972524 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" event={"ID":"414b332d-83f8-4e88-97de-156170e1cba9","Type":"ContainerStarted","Data":"9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78"} Oct 31 00:40:16.972970 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:16.972860 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:40:16.972970 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:16.972919 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:40:16.972970 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:16.972952 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:40:16.974604 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:16.974559 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:40:16.975084 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:16.975059 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:40:16.977948 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:16.977928 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:40:16.996977 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:16.996381 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podStartSLOduration=1.947163578 podStartE2EDuration="42.996364957s" totalImagesPullingTime="36.913821999s" totalInitContainerRuntime="4.13537938s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:16.994799166 +0000 UTC m=+600.356381806" watchObservedRunningTime="2026-10-31 00:40:16.996364957 +0000 UTC m=+600.357947597" Oct 31 00:40:17.976137 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:17.976088 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:40:17.976547 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:17.976300 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:40:18.979735 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:18.979698 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:40:18.980196 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:18.980083 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:40:19.840147 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:40:19.840044 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:40:25.955123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:25.955082 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused" Oct 31 00:40:25.955516 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:25.955082 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused" Oct 31 00:40:28.980133 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:28.980079 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:40:28.980591 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:28.980518 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:40:35.954300 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:35.954258 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused" Oct 31 00:40:35.954747 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:35.954410 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused" Oct 31 00:40:38.980514 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:38.980468 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:40:38.981024 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:38.981002 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:40:45.954400 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:45.954355 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused" Oct 31 00:40:45.954400 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:45.954374 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused" Oct 31 00:40:48.980533 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:48.980494 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:40:48.981052 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:48.980965 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:40:49.840303 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:40:49.840266 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:40:55.954398 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:55.954357 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused" Oct 31 00:40:55.954819 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:55.954357 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused" Oct 31 00:40:58.980390 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:58.980289 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:40:58.980828 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:40:58.980741 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:05.954552 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:05.954516 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused" Oct 31 00:41:05.954989 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:05.954520 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused" Oct 31 00:41:08.980100 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:08.980047 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:41:08.980571 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:08.980452 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:15.954738 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:15.954700 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused" Oct 31 00:41:15.955163 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:15.954701 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused" Oct 31 00:41:18.980519 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:18.980473 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:41:18.981020 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:18.980948 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:19.838551 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:41:19.838518 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:41:25.772891 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:25.772853 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:41:25.773374 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:25.773267 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:25.954747 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:25.954710 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused" Oct 31 00:41:25.954910 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:25.954710 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused" Oct 31 00:41:32.777395 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:32.777367 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:41:32.946502 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:32.946467 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk"] Oct 31 00:41:32.946771 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:32.946715 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" containerID="cri-o://c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc" gracePeriod=30 Oct 31 00:41:32.946931 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:32.946779 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kube-rbac-proxy" containerID="cri-o://c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d" gracePeriod=30 Oct 31 00:41:32.947827 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:32.947652 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:41:33.249915 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:33.249867 5821 generic.go:356] "Generic (PLEG): container finished" podID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerID="c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d" exitCode=0 Oct 31 00:41:33.250076 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:33.249940 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" event={"ID":"2ce13928-1a73-4459-8a6b-8e37e8d2160e","Type":"ContainerDied","Data":"c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d"} Oct 31 00:41:34.025466 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.025432 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr"] Oct 31 00:41:34.091021 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.090986 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs"] Oct 31 00:41:34.094376 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.094354 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.103507 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.103480 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs"] Oct 31 00:41:34.205299 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.205268 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q55z\" (UniqueName: \"kubernetes.io/projected/9fa8f943-8879-473d-a84f-f907645afaf3-kube-api-access-5q55z\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.205299 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.205303 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fa8f943-8879-473d-a84f-f907645afaf3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.205536 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.205345 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fa8f943-8879-473d-a84f-f907645afaf3-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.205536 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.205492 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.205536 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.205522 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.253541 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.253500 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" containerID="cri-o://2eec6c1e0ac0719ab24fa169be71cb41ef090107d847a87f0e2b661bd2a417e4" gracePeriod=30 Oct 31 00:41:34.253541 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.253517 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kube-rbac-proxy" containerID="cri-o://394b7b3ad9fc7aa121d3102f2f3361621cd6f05310c9286167f47f99ee843aef" gracePeriod=30 Oct 31 00:41:34.306074 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.305986 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fa8f943-8879-473d-a84f-f907645afaf3-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.306074 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.306041 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.306074 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.306070 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.306360 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.306128 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q55z\" (UniqueName: \"kubernetes.io/projected/9fa8f943-8879-473d-a84f-f907645afaf3-kube-api-access-5q55z\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.306360 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.306152 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fa8f943-8879-473d-a84f-f907645afaf3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.306576 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.306552 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fa8f943-8879-473d-a84f-f907645afaf3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.306839 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.306815 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.306901 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.306820 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.308832 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.308813 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fa8f943-8879-473d-a84f-f907645afaf3-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.315651 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.315598 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q55z\" (UniqueName: \"kubernetes.io/projected/9fa8f943-8879-473d-a84f-f907645afaf3-kube-api-access-5q55z\") pod \"isvc-canary-rollback-predictor-5bb5f8559f-4krqs\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.412093 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.412062 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:34.682798 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:34.682768 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs"] Oct 31 00:41:34.685640 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:41:34.685594 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa8f943_8879_473d_a84f_f907645afaf3.slice/crio-e96d0ee5ce8a68dc8f7c450dc8cf6b3d87ee4181fef9d50508983661268a45ee WatchSource:0}: Error finding container e96d0ee5ce8a68dc8f7c450dc8cf6b3d87ee4181fef9d50508983661268a45ee: Status 404 returned error can't find the container with id e96d0ee5ce8a68dc8f7c450dc8cf6b3d87ee4181fef9d50508983661268a45ee Oct 31 00:41:35.259304 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:35.259268 5821 generic.go:356] "Generic (PLEG): container finished" podID="9f0b8382-f79e-46f2-986c-4a1680707822" containerID="394b7b3ad9fc7aa121d3102f2f3361621cd6f05310c9286167f47f99ee843aef" exitCode=0 Oct 31 00:41:35.259763 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:35.259339 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" event={"ID":"9f0b8382-f79e-46f2-986c-4a1680707822","Type":"ContainerDied","Data":"394b7b3ad9fc7aa121d3102f2f3361621cd6f05310c9286167f47f99ee843aef"} Oct 31 00:41:35.261111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:35.261084 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" event={"ID":"9fa8f943-8879-473d-a84f-f907645afaf3","Type":"ContainerStarted","Data":"3f041f067f77af115e4cbc24ff62aedaec19be72ad5d2ebb21efc40d396edeaa"} Oct 31 00:41:35.261234 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:35.261146 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" event={"ID":"9fa8f943-8879-473d-a84f-f907645afaf3","Type":"ContainerStarted","Data":"e96d0ee5ce8a68dc8f7c450dc8cf6b3d87ee4181fef9d50508983661268a45ee"} Oct 31 00:41:35.774342 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:35.774302 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:41:35.774645 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:35.774471 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:41:35.948461 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:35.948421 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused" Oct 31 00:41:35.948647 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:35.948421 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.17:8643/healthz\": dial tcp 10.134.0.17:8643: connect: connection refused" Oct 31 00:41:38.192277 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.192246 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:41:38.273897 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.273802 5821 generic.go:356] "Generic (PLEG): container finished" podID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerID="c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc" exitCode=0 Oct 31 00:41:38.273897 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.273831 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" event={"ID":"2ce13928-1a73-4459-8a6b-8e37e8d2160e","Type":"ContainerDied","Data":"c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc"} Oct 31 00:41:38.273897 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.273877 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" Oct 31 00:41:38.274129 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.273907 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk" event={"ID":"2ce13928-1a73-4459-8a6b-8e37e8d2160e","Type":"ContainerDied","Data":"ddd957a278d41f769f95e60ad2a6f3457636c712910294f53b81cda9c88dc3ae"} Oct 31 00:41:38.274129 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.273944 5821 scope.go:122] "RemoveContainer" containerID="c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d" Oct 31 00:41:38.281800 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.281785 5821 scope.go:122] "RemoveContainer" containerID="c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc" Oct 31 00:41:38.289215 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.289200 5821 scope.go:122] "RemoveContainer" containerID="8f9ff64f0a03fe52388c98970f5dd45ad98694f3a21abfafbfdab67b8c562e48" Oct 31 00:41:38.296429 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.296407 5821 scope.go:122] "RemoveContainer" containerID="c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d" Oct 31 00:41:38.296762 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:41:38.296737 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d\": container with ID starting with c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d not found: ID does not exist" containerID="c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d" Oct 31 00:41:38.296837 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.296773 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d"} err="failed to get container status \"c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d\": rpc error: code = NotFound desc = could not find container \"c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d\": container with ID starting with c4849b57440c6e260582373474f8b6b5e7f18e9abd5e05d401810faa4bd8727d not found: ID does not exist" Oct 31 00:41:38.296837 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.296797 5821 scope.go:122] "RemoveContainer" containerID="c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc" Oct 31 00:41:38.297085 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:41:38.297067 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc\": container with ID starting with c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc not found: ID does not exist" containerID="c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc" Oct 31 00:41:38.297131 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.297094 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc"} err="failed to get container status \"c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc\": rpc error: code = NotFound desc = could not find container \"c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc\": container with ID starting with c6044fcc32970e12c0931c8dadfa8873d1812e8055d6cd0c057d703b1a0358fc not found: ID does not exist" Oct 31 00:41:38.297131 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.297109 5821 scope.go:122] "RemoveContainer" containerID="8f9ff64f0a03fe52388c98970f5dd45ad98694f3a21abfafbfdab67b8c562e48" Oct 31 00:41:38.297393 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:41:38.297378 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9ff64f0a03fe52388c98970f5dd45ad98694f3a21abfafbfdab67b8c562e48\": container with ID starting with 8f9ff64f0a03fe52388c98970f5dd45ad98694f3a21abfafbfdab67b8c562e48 not found: ID does not exist" containerID="8f9ff64f0a03fe52388c98970f5dd45ad98694f3a21abfafbfdab67b8c562e48" Oct 31 00:41:38.297541 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.297399 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9ff64f0a03fe52388c98970f5dd45ad98694f3a21abfafbfdab67b8c562e48"} err="failed to get container status \"8f9ff64f0a03fe52388c98970f5dd45ad98694f3a21abfafbfdab67b8c562e48\": rpc error: code = NotFound desc = could not find container \"8f9ff64f0a03fe52388c98970f5dd45ad98694f3a21abfafbfdab67b8c562e48\": container with ID starting with 8f9ff64f0a03fe52388c98970f5dd45ad98694f3a21abfafbfdab67b8c562e48 not found: ID does not exist" Oct 31 00:41:38.340417 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.340385 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-cabundle-cert\") pod \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " Oct 31 00:41:38.340603 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.340456 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-rbac-proxy-sar-config\") pod \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " Oct 31 00:41:38.340603 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.340502 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ce13928-1a73-4459-8a6b-8e37e8d2160e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ce13928-1a73-4459-8a6b-8e37e8d2160e-proxy-tls\") pod \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " Oct 31 00:41:38.340603 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.340530 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kserve-provision-location\") pod \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " Oct 31 00:41:38.340603 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.340598 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-api-access-zgxrv\" (UniqueName: \"kubernetes.io/projected/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-api-access-zgxrv\") pod \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\" (UID: \"2ce13928-1a73-4459-8a6b-8e37e8d2160e\") " Oct 31 00:41:38.340868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.340839 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kserve-provision-location" pod "2ce13928-1a73-4459-8a6b-8e37e8d2160e" (UID: "2ce13928-1a73-4459-8a6b-8e37e8d2160e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:38.340868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.340859 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-cabundle-cert" pod "2ce13928-1a73-4459-8a6b-8e37e8d2160e" (UID: "2ce13928-1a73-4459-8a6b-8e37e8d2160e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:38.340954 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.340881 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-rbac-proxy-sar-config" pod "2ce13928-1a73-4459-8a6b-8e37e8d2160e" (UID: "2ce13928-1a73-4459-8a6b-8e37e8d2160e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:38.340991 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.340981 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:38.341032 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.340998 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:38.341032 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.341012 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:38.343028 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.343005 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ce13928-1a73-4459-8a6b-8e37e8d2160e-proxy-tls" pod "2ce13928-1a73-4459-8a6b-8e37e8d2160e" (UID: "2ce13928-1a73-4459-8a6b-8e37e8d2160e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:38.343148 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.343108 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-api-access-zgxrv" pod "2ce13928-1a73-4459-8a6b-8e37e8d2160e" (UID: "2ce13928-1a73-4459-8a6b-8e37e8d2160e"). InnerVolumeSpecName "kube-api-access-zgxrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:38.441532 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.441494 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ce13928-1a73-4459-8a6b-8e37e8d2160e-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:38.441532 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.441526 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgxrv\" (UniqueName: \"kubernetes.io/projected/2ce13928-1a73-4459-8a6b-8e37e8d2160e-kube-api-access-zgxrv\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:38.596506 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.596473 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk"] Oct 31 00:41:38.600922 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.600897 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-bzszk"] Oct 31 00:41:38.777531 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:38.777498 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" path="/var/lib/kubelet/pods/2ce13928-1a73-4459-8a6b-8e37e8d2160e/volumes" Oct 31 00:41:39.281264 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.281229 5821 generic.go:356] "Generic (PLEG): container finished" podID="9f0b8382-f79e-46f2-986c-4a1680707822" containerID="2eec6c1e0ac0719ab24fa169be71cb41ef090107d847a87f0e2b661bd2a417e4" exitCode=0 Oct 31 00:41:39.281665 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.281291 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" event={"ID":"9f0b8382-f79e-46f2-986c-4a1680707822","Type":"ContainerDied","Data":"2eec6c1e0ac0719ab24fa169be71cb41ef090107d847a87f0e2b661bd2a417e4"} Oct 31 00:41:39.284047 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.284028 5821 generic.go:356] "Generic (PLEG): container finished" podID="9fa8f943-8879-473d-a84f-f907645afaf3" containerID="3f041f067f77af115e4cbc24ff62aedaec19be72ad5d2ebb21efc40d396edeaa" exitCode=0 Oct 31 00:41:39.284142 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.284086 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" event={"ID":"9fa8f943-8879-473d-a84f-f907645afaf3","Type":"ContainerDied","Data":"3f041f067f77af115e4cbc24ff62aedaec19be72ad5d2ebb21efc40d396edeaa"} Oct 31 00:41:39.403541 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.403520 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:41:39.550923 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.550840 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-kube-rbac-proxy-sar-config\") pod \"9f0b8382-f79e-46f2-986c-4a1680707822\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " Oct 31 00:41:39.550923 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.550904 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0b8382-f79e-46f2-986c-4a1680707822-kube-api-access-snpzj\" (UniqueName: \"kubernetes.io/projected/9f0b8382-f79e-46f2-986c-4a1680707822-kube-api-access-snpzj\") pod \"9f0b8382-f79e-46f2-986c-4a1680707822\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " Oct 31 00:41:39.551163 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.550928 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-cabundle-cert\") pod \"9f0b8382-f79e-46f2-986c-4a1680707822\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " Oct 31 00:41:39.551163 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.550955 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f0b8382-f79e-46f2-986c-4a1680707822-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0b8382-f79e-46f2-986c-4a1680707822-proxy-tls\") pod \"9f0b8382-f79e-46f2-986c-4a1680707822\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " Oct 31 00:41:39.551163 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.550988 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0b8382-f79e-46f2-986c-4a1680707822-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f0b8382-f79e-46f2-986c-4a1680707822-kserve-provision-location\") pod \"9f0b8382-f79e-46f2-986c-4a1680707822\" (UID: \"9f0b8382-f79e-46f2-986c-4a1680707822\") " Oct 31 00:41:39.551323 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.551286 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-kube-rbac-proxy-sar-config" pod "9f0b8382-f79e-46f2-986c-4a1680707822" (UID: "9f0b8382-f79e-46f2-986c-4a1680707822"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:39.551381 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.551361 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0b8382-f79e-46f2-986c-4a1680707822-kserve-provision-location" pod "9f0b8382-f79e-46f2-986c-4a1680707822" (UID: "9f0b8382-f79e-46f2-986c-4a1680707822"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:39.551381 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.551368 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-cabundle-cert" pod "9f0b8382-f79e-46f2-986c-4a1680707822" (UID: "9f0b8382-f79e-46f2-986c-4a1680707822"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:39.553304 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.553283 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0b8382-f79e-46f2-986c-4a1680707822-kube-api-access-snpzj" pod "9f0b8382-f79e-46f2-986c-4a1680707822" (UID: "9f0b8382-f79e-46f2-986c-4a1680707822"). InnerVolumeSpecName "kube-api-access-snpzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:39.553304 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.553289 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0b8382-f79e-46f2-986c-4a1680707822-proxy-tls" pod "9f0b8382-f79e-46f2-986c-4a1680707822" (UID: "9f0b8382-f79e-46f2-986c-4a1680707822"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:39.652150 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.652117 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snpzj\" (UniqueName: \"kubernetes.io/projected/9f0b8382-f79e-46f2-986c-4a1680707822-kube-api-access-snpzj\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:39.652150 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.652145 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:39.652150 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.652156 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0b8382-f79e-46f2-986c-4a1680707822-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:39.652369 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.652166 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f0b8382-f79e-46f2-986c-4a1680707822-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:39.652369 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:39.652176 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0b8382-f79e-46f2-986c-4a1680707822-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:40.289562 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.289473 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" event={"ID":"9fa8f943-8879-473d-a84f-f907645afaf3","Type":"ContainerStarted","Data":"971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6"} Oct 31 00:41:40.289562 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.289558 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" event={"ID":"9fa8f943-8879-473d-a84f-f907645afaf3","Type":"ContainerStarted","Data":"9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc"} Oct 31 00:41:40.290052 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.289975 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:40.290154 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.290135 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:40.296293 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.295783 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Oct 31 00:41:40.298256 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.298224 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" event={"ID":"9f0b8382-f79e-46f2-986c-4a1680707822","Type":"ContainerDied","Data":"1844d664f217d27e10e4636852e2fdce4c36770c21a69a438a425db96b16b098"} Oct 31 00:41:40.298401 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.298264 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr" Oct 31 00:41:40.298401 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.298312 5821 scope.go:122] "RemoveContainer" containerID="394b7b3ad9fc7aa121d3102f2f3361621cd6f05310c9286167f47f99ee843aef" Oct 31 00:41:40.307157 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.307129 5821 scope.go:122] "RemoveContainer" containerID="2eec6c1e0ac0719ab24fa169be71cb41ef090107d847a87f0e2b661bd2a417e4" Oct 31 00:41:40.311968 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.311912 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podStartSLOduration=2.715904418 podStartE2EDuration="6.31189337s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.595988952s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:40.310489272 +0000 UTC m=+683.672071910" watchObservedRunningTime="2026-10-31 00:41:40.31189337 +0000 UTC m=+683.673476013" Oct 31 00:41:40.315755 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.315738 5821 scope.go:122] "RemoveContainer" containerID="300b426e005f70d89a3cc6679a1bcd8511008288b953874db522fbfdd8a3eb51" Oct 31 00:41:40.324431 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.324408 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr"] Oct 31 00:41:40.329139 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.329118 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-v2-predictor-785fd7445d-jl5kr"] Oct 31 00:41:40.778084 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:40.778052 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" path="/var/lib/kubelet/pods/9f0b8382-f79e-46f2-986c-4a1680707822/volumes" Oct 31 00:41:41.303876 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:41.303840 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Oct 31 00:41:46.308713 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:46.308686 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:41:46.309204 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:46.309176 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Oct 31 00:41:49.727936 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:49.727308 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t"] Oct 31 00:41:49.727936 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:49.727615 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" containerID="cri-o://8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570" gracePeriod=30 Oct 31 00:41:49.728479 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:49.727985 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" containerID="cri-o://9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78" gracePeriod=30 Oct 31 00:41:49.728479 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:49.728083 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kube-rbac-proxy" containerID="cri-o://d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308" gracePeriod=30 Oct 31 00:41:49.842111 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:41:49.842077 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:41:50.337042 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:50.337012 5821 generic.go:356] "Generic (PLEG): container finished" podID="414b332d-83f8-4e88-97de-156170e1cba9" containerID="d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308" exitCode=0 Oct 31 00:41:50.337226 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:50.337094 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" event={"ID":"414b332d-83f8-4e88-97de-156170e1cba9","Type":"ContainerDied","Data":"d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308"} Oct 31 00:41:51.973495 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:51.973456 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused" Oct 31 00:41:55.358982 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:55.358945 5821 generic.go:356] "Generic (PLEG): container finished" podID="414b332d-83f8-4e88-97de-156170e1cba9" containerID="8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570" exitCode=0 Oct 31 00:41:55.359365 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:55.359004 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" event={"ID":"414b332d-83f8-4e88-97de-156170e1cba9","Type":"ContainerDied","Data":"8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570"} Oct 31 00:41:55.773273 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:55.773224 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:41:55.773558 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:55.773528 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:56.309193 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:56.309154 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Oct 31 00:41:56.973330 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:41:56.973292 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused" Oct 31 00:42:01.973501 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:01.973458 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused" Oct 31 00:42:01.973917 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:01.973603 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:42:05.773050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:05.773007 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:42:05.773501 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:05.773288 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:06.309611 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:06.309566 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Oct 31 00:42:06.973084 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:06.973039 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused" Oct 31 00:42:11.973096 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:11.973056 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused" Oct 31 00:42:15.772877 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:15.772836 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused" Oct 31 00:42:15.773313 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:15.773010 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:42:15.773313 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:15.773149 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:15.773313 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:15.773254 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:42:16.309177 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:16.309144 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Oct 31 00:42:16.973020 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:16.972977 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused" Oct 31 00:42:19.837064 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:42:19.837027 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:42:20.374791 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.374766 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:42:20.447258 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.447229 5821 generic.go:356] "Generic (PLEG): container finished" podID="414b332d-83f8-4e88-97de-156170e1cba9" containerID="9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78" exitCode=0 Oct 31 00:42:20.447444 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.447291 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" Oct 31 00:42:20.447444 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.447326 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" event={"ID":"414b332d-83f8-4e88-97de-156170e1cba9","Type":"ContainerDied","Data":"9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78"} Oct 31 00:42:20.447444 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.447405 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t" event={"ID":"414b332d-83f8-4e88-97de-156170e1cba9","Type":"ContainerDied","Data":"5d3ba1d95efc742b1f079e953acc18ad39e04cfb8e992f8627515b2b37156b5a"} Oct 31 00:42:20.447637 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.447450 5821 scope.go:122] "RemoveContainer" containerID="9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78" Oct 31 00:42:20.455266 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.455250 5821 scope.go:122] "RemoveContainer" containerID="d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308" Oct 31 00:42:20.462298 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.462282 5821 scope.go:122] "RemoveContainer" containerID="8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570" Oct 31 00:42:20.468360 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.468340 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-cabundle-cert\") pod \"414b332d-83f8-4e88-97de-156170e1cba9\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " Oct 31 00:42:20.468454 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.468376 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414b332d-83f8-4e88-97de-156170e1cba9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/414b332d-83f8-4e88-97de-156170e1cba9-kserve-provision-location\") pod \"414b332d-83f8-4e88-97de-156170e1cba9\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " Oct 31 00:42:20.468454 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.468448 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414b332d-83f8-4e88-97de-156170e1cba9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/414b332d-83f8-4e88-97de-156170e1cba9-proxy-tls\") pod \"414b332d-83f8-4e88-97de-156170e1cba9\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " Oct 31 00:42:20.468568 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.468478 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414b332d-83f8-4e88-97de-156170e1cba9-kube-api-access-z9jpl\" (UniqueName: \"kubernetes.io/projected/414b332d-83f8-4e88-97de-156170e1cba9-kube-api-access-z9jpl\") pod \"414b332d-83f8-4e88-97de-156170e1cba9\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " Oct 31 00:42:20.468568 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.468497 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-kube-rbac-proxy-sar-config\") pod \"414b332d-83f8-4e88-97de-156170e1cba9\" (UID: \"414b332d-83f8-4e88-97de-156170e1cba9\") " Oct 31 00:42:20.468785 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.468765 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-cabundle-cert" pod "414b332d-83f8-4e88-97de-156170e1cba9" (UID: "414b332d-83f8-4e88-97de-156170e1cba9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:20.468785 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.468773 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414b332d-83f8-4e88-97de-156170e1cba9-kserve-provision-location" pod "414b332d-83f8-4e88-97de-156170e1cba9" (UID: "414b332d-83f8-4e88-97de-156170e1cba9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:20.468957 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.468939 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-kube-rbac-proxy-sar-config" pod "414b332d-83f8-4e88-97de-156170e1cba9" (UID: "414b332d-83f8-4e88-97de-156170e1cba9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:20.469456 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.469437 5821 scope.go:122] "RemoveContainer" containerID="3aa629e43f845af946b82defe14cdf9a2a106ecf15ddcb13e7a063db7ad562b3" Oct 31 00:42:20.470912 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.470888 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414b332d-83f8-4e88-97de-156170e1cba9-kube-api-access-z9jpl" pod "414b332d-83f8-4e88-97de-156170e1cba9" (UID: "414b332d-83f8-4e88-97de-156170e1cba9"). InnerVolumeSpecName "kube-api-access-z9jpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:20.471011 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.470931 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414b332d-83f8-4e88-97de-156170e1cba9-proxy-tls" pod "414b332d-83f8-4e88-97de-156170e1cba9" (UID: "414b332d-83f8-4e88-97de-156170e1cba9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:20.483158 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.483131 5821 scope.go:122] "RemoveContainer" containerID="9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78" Oct 31 00:42:20.483683 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:42:20.483542 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78\": container with ID starting with 9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78 not found: ID does not exist" containerID="9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78" Oct 31 00:42:20.483683 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.483579 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78"} err="failed to get container status \"9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78\": rpc error: code = NotFound desc = could not find container \"9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78\": container with ID starting with 9ab007270b82ad834de208fbc3d5818ac3d433b5a9c0bb81526aa091e9838b78 not found: ID does not exist" Oct 31 00:42:20.483683 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.483602 5821 scope.go:122] "RemoveContainer" containerID="d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308" Oct 31 00:42:20.484768 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:42:20.484731 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308\": container with ID starting with d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308 not found: ID does not exist" containerID="d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308" Oct 31 00:42:20.484867 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.484765 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308"} err="failed to get container status \"d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308\": rpc error: code = NotFound desc = could not find container \"d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308\": container with ID starting with d12fc6c124e9b90fc52eb9a90095e72e2ee0b033c4369bd3474e8196159bf308 not found: ID does not exist" Oct 31 00:42:20.484867 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.484785 5821 scope.go:122] "RemoveContainer" containerID="8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570" Oct 31 00:42:20.485792 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:42:20.485735 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570\": container with ID starting with 8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570 not found: ID does not exist" containerID="8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570" Oct 31 00:42:20.485792 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.485766 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570"} err="failed to get container status \"8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570\": rpc error: code = NotFound desc = could not find container \"8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570\": container with ID starting with 8247db0ff000f9ac3adfa969a27d577828015fc2328a331e48cfa9219011d570 not found: ID does not exist" Oct 31 00:42:20.485792 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.485783 5821 scope.go:122] "RemoveContainer" containerID="3aa629e43f845af946b82defe14cdf9a2a106ecf15ddcb13e7a063db7ad562b3" Oct 31 00:42:20.486333 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:42:20.486283 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa629e43f845af946b82defe14cdf9a2a106ecf15ddcb13e7a063db7ad562b3\": container with ID starting with 3aa629e43f845af946b82defe14cdf9a2a106ecf15ddcb13e7a063db7ad562b3 not found: ID does not exist" containerID="3aa629e43f845af946b82defe14cdf9a2a106ecf15ddcb13e7a063db7ad562b3" Oct 31 00:42:20.486333 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.486312 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa629e43f845af946b82defe14cdf9a2a106ecf15ddcb13e7a063db7ad562b3"} err="failed to get container status \"3aa629e43f845af946b82defe14cdf9a2a106ecf15ddcb13e7a063db7ad562b3\": rpc error: code = NotFound desc = could not find container \"3aa629e43f845af946b82defe14cdf9a2a106ecf15ddcb13e7a063db7ad562b3\": container with ID starting with 3aa629e43f845af946b82defe14cdf9a2a106ecf15ddcb13e7a063db7ad562b3 not found: ID does not exist" Oct 31 00:42:20.569698 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.569601 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/414b332d-83f8-4e88-97de-156170e1cba9-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:20.569698 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.569650 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9jpl\" (UniqueName: \"kubernetes.io/projected/414b332d-83f8-4e88-97de-156170e1cba9-kube-api-access-z9jpl\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:20.569698 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.569666 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:20.569698 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.569679 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/414b332d-83f8-4e88-97de-156170e1cba9-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:20.569698 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.569693 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/414b332d-83f8-4e88-97de-156170e1cba9-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:20.768754 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.768722 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t"] Oct 31 00:42:20.777469 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:20.777447 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-raw-sklearn-batcher-757b4-predictor-7f689c549d-mh68t"] Oct 31 00:42:21.587285 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587244 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b"] Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587554 5821 cpu_manager.go:414] "containerMap: removing container" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587565 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587578 5821 cpu_manager.go:414] "containerMap: removing container" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587583 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587592 5821 cpu_manager.go:414] "containerMap: removing container" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="storage-initializer" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587600 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="storage-initializer" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587606 5821 cpu_manager.go:414] "containerMap: removing container" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kube-rbac-proxy" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587611 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kube-rbac-proxy" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587635 5821 cpu_manager.go:414] "containerMap: removing container" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kube-rbac-proxy" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587644 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kube-rbac-proxy" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587660 5821 cpu_manager.go:414] "containerMap: removing container" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587667 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587679 5821 cpu_manager.go:414] "containerMap: removing container" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587686 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587695 5821 cpu_manager.go:414] "containerMap: removing container" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kube-rbac-proxy" Oct 31 00:42:21.587692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587700 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kube-rbac-proxy" Oct 31 00:42:21.588175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587711 5821 cpu_manager.go:414] "containerMap: removing container" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="storage-initializer" Oct 31 00:42:21.588175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587716 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="storage-initializer" Oct 31 00:42:21.588175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587725 5821 cpu_manager.go:414] "containerMap: removing container" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="storage-initializer" Oct 31 00:42:21.588175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587729 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="storage-initializer" Oct 31 00:42:21.588175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587790 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kserve-container" Oct 31 00:42:21.588175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587798 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kserve-container" Oct 31 00:42:21.588175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587804 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="kube-rbac-proxy" Oct 31 00:42:21.588175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587811 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="414b332d-83f8-4e88-97de-156170e1cba9" containerName="agent" Oct 31 00:42:21.588175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587817 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kserve-container" Oct 31 00:42:21.588175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587822 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0b8382-f79e-46f2-986c-4a1680707822" containerName="kube-rbac-proxy" Oct 31 00:42:21.588175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.587828 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce13928-1a73-4459-8a6b-8e37e8d2160e" containerName="kube-rbac-proxy" Oct 31 00:42:21.597848 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.597822 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.602253 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.602159 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b"] Oct 31 00:42:21.602395 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.602276 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"openshift-service-ca.crt\"" Oct 31 00:42:21.602395 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.602381 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-sklearn-graph-raw-ad17a-kube-rbac-proxy-sar-config\"" Oct 31 00:42:21.602505 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.602414 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:42:21.602505 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.602475 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"default-dockercfg-gs5fn\"" Oct 31 00:42:21.602602 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.602535 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"seaweedfs-s3-creds\"" Oct 31 00:42:21.602602 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.602534 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"kube-root-ca.crt\"" Oct 31 00:42:21.602602 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.602533 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-sklearn-graph-raw-ad17a-predictor-serving-cert\"" Oct 31 00:42:21.679972 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.679945 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.679972 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.679977 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39731e33-baa9-4164-881d-9dc637c3281b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.680209 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.680011 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxp9n\" (UniqueName: \"kubernetes.io/projected/39731e33-baa9-4164-881d-9dc637c3281b-kube-api-access-rxp9n\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.680209 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.680113 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39731e33-baa9-4164-881d-9dc637c3281b-proxy-tls\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.680209 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.680162 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.781144 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.781101 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxp9n\" (UniqueName: \"kubernetes.io/projected/39731e33-baa9-4164-881d-9dc637c3281b-kube-api-access-rxp9n\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.781332 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.781154 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39731e33-baa9-4164-881d-9dc637c3281b-proxy-tls\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.781332 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.781198 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.781332 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.781239 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.781332 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.781264 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39731e33-baa9-4164-881d-9dc637c3281b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.781791 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.781768 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39731e33-baa9-4164-881d-9dc637c3281b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.781994 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.781973 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.782038 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.782003 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.783761 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.783744 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39731e33-baa9-4164-881d-9dc637c3281b-proxy-tls\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.788394 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.788371 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx"] Oct 31 00:42:21.792077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.792055 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.793713 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.793692 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxp9n\" (UniqueName: \"kubernetes.io/projected/39731e33-baa9-4164-881d-9dc637c3281b-kube-api-access-rxp9n\") pod \"isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.800696 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.800657 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-xgboost-graph-raw-ad17a-kube-rbac-proxy-sar-config\"" Oct 31 00:42:21.800839 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.800739 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-xgboost-graph-raw-ad17a-predictor-serving-cert\"" Oct 31 00:42:21.801777 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.801755 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx"] Oct 31 00:42:21.882721 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.882605 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/960da667-4f1b-45a2-ac83-c664f62aca2c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.882844 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.882731 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmqml\" (UniqueName: \"kubernetes.io/projected/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-api-access-tmqml\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.882844 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.882757 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960da667-4f1b-45a2-ac83-c664f62aca2c-proxy-tls\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.882921 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.882841 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.882921 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.882879 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.916196 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.916169 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:21.983460 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.983427 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.983642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.983487 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/960da667-4f1b-45a2-ac83-c664f62aca2c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.983642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.983521 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmqml\" (UniqueName: \"kubernetes.io/projected/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-api-access-tmqml\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.983642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.983544 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960da667-4f1b-45a2-ac83-c664f62aca2c-proxy-tls\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.983642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.983568 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.984012 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.983991 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/960da667-4f1b-45a2-ac83-c664f62aca2c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.984217 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.984195 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.984367 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.984344 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.986275 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.986260 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960da667-4f1b-45a2-ac83-c664f62aca2c-proxy-tls\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:21.992074 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:21.992053 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmqml\" (UniqueName: \"kubernetes.io/projected/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-api-access-tmqml\") pod \"isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:22.115943 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:22.115912 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:22.190890 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:22.190859 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b"] Oct 31 00:42:22.193428 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:42:22.193401 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39731e33_baa9_4164_881d_9dc637c3281b.slice/crio-d1de3fd1eb75eb8c05c82dd196ac070d0e7e8abe12cac057ca101ec2e51ee030 WatchSource:0}: Error finding container d1de3fd1eb75eb8c05c82dd196ac070d0e7e8abe12cac057ca101ec2e51ee030: Status 404 returned error can't find the container with id d1de3fd1eb75eb8c05c82dd196ac070d0e7e8abe12cac057ca101ec2e51ee030 Oct 31 00:42:22.388201 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:22.388164 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx"] Oct 31 00:42:22.391458 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:42:22.391430 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960da667_4f1b_45a2_ac83_c664f62aca2c.slice/crio-272f3af34cfc09f0538b3e3aa4dd0a278cab9e6d8348d451fe1be9a754ec69eb WatchSource:0}: Error finding container 272f3af34cfc09f0538b3e3aa4dd0a278cab9e6d8348d451fe1be9a754ec69eb: Status 404 returned error can't find the container with id 272f3af34cfc09f0538b3e3aa4dd0a278cab9e6d8348d451fe1be9a754ec69eb Oct 31 00:42:22.457091 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:22.457057 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" event={"ID":"960da667-4f1b-45a2-ac83-c664f62aca2c","Type":"ContainerStarted","Data":"d200e8c4ca40f6134890605a67977ccec0314ed124d1998c289ece4dac5f5372"} Oct 31 00:42:22.457236 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:22.457130 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" event={"ID":"960da667-4f1b-45a2-ac83-c664f62aca2c","Type":"ContainerStarted","Data":"272f3af34cfc09f0538b3e3aa4dd0a278cab9e6d8348d451fe1be9a754ec69eb"} Oct 31 00:42:22.458837 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:22.458804 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" event={"ID":"39731e33-baa9-4164-881d-9dc637c3281b","Type":"ContainerStarted","Data":"ee815cbcfdb4deff2dfd9359feeb3e83f0f1d969b2eac88c80e67a771ad77553"} Oct 31 00:42:22.458945 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:22.458853 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" event={"ID":"39731e33-baa9-4164-881d-9dc637c3281b","Type":"ContainerStarted","Data":"d1de3fd1eb75eb8c05c82dd196ac070d0e7e8abe12cac057ca101ec2e51ee030"} Oct 31 00:42:22.778956 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:22.778877 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414b332d-83f8-4e88-97de-156170e1cba9" path="/var/lib/kubelet/pods/414b332d-83f8-4e88-97de-156170e1cba9/volumes" Oct 31 00:42:26.309261 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:26.309188 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Oct 31 00:42:26.475828 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:26.475798 5821 generic.go:356] "Generic (PLEG): container finished" podID="39731e33-baa9-4164-881d-9dc637c3281b" containerID="ee815cbcfdb4deff2dfd9359feeb3e83f0f1d969b2eac88c80e67a771ad77553" exitCode=0 Oct 31 00:42:26.475981 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:26.475875 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" event={"ID":"39731e33-baa9-4164-881d-9dc637c3281b","Type":"ContainerDied","Data":"ee815cbcfdb4deff2dfd9359feeb3e83f0f1d969b2eac88c80e67a771ad77553"} Oct 31 00:42:26.477464 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:26.477444 5821 generic.go:356] "Generic (PLEG): container finished" podID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerID="d200e8c4ca40f6134890605a67977ccec0314ed124d1998c289ece4dac5f5372" exitCode=0 Oct 31 00:42:26.477584 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:26.477495 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" event={"ID":"960da667-4f1b-45a2-ac83-c664f62aca2c","Type":"ContainerDied","Data":"d200e8c4ca40f6134890605a67977ccec0314ed124d1998c289ece4dac5f5372"} Oct 31 00:42:26.478841 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:26.478794 5821 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:42:27.485279 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:27.485240 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" event={"ID":"39731e33-baa9-4164-881d-9dc637c3281b","Type":"ContainerStarted","Data":"dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183"} Oct 31 00:42:27.485779 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:27.485319 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" event={"ID":"39731e33-baa9-4164-881d-9dc637c3281b","Type":"ContainerStarted","Data":"44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49"} Oct 31 00:42:27.485779 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:27.485676 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:27.485779 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:27.485717 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:27.487192 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:27.487170 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 31 00:42:27.505707 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:27.505650 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podStartSLOduration=2.695161596 podStartE2EDuration="6.505633165s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.810471569s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:27.504725892 +0000 UTC m=+730.866308532" watchObservedRunningTime="2026-10-31 00:42:27.505633165 +0000 UTC m=+730.867215798" Oct 31 00:42:28.492015 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:28.491969 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 31 00:42:33.498928 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:33.498894 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:42:33.499860 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:33.499820 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 31 00:42:36.309773 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:36.309739 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Oct 31 00:42:43.500864 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:43.500821 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 31 00:42:45.569132 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:45.569098 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" event={"ID":"960da667-4f1b-45a2-ac83-c664f62aca2c","Type":"ContainerStarted","Data":"acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b"} Oct 31 00:42:45.569489 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:45.569184 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" event={"ID":"960da667-4f1b-45a2-ac83-c664f62aca2c","Type":"ContainerStarted","Data":"899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9"} Oct 31 00:42:45.569489 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:45.569376 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:45.591827 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:45.591767 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podStartSLOduration=1.988899596 podStartE2EDuration="24.591751322s" totalImagesPullingTime="18.655859856s" totalInitContainerRuntime="3.94699187s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:45.589839615 +0000 UTC m=+748.951422267" watchObservedRunningTime="2026-10-31 00:42:45.591751322 +0000 UTC m=+748.953333961" Oct 31 00:42:46.309257 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:46.309220 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Oct 31 00:42:46.573102 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:46.573019 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:46.574392 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:46.574367 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 31 00:42:47.577118 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:47.577080 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 31 00:42:49.839239 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:42:49.839204 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:42:52.581958 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:52.581929 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:42:52.582439 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:52.582414 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 31 00:42:53.500718 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:53.500677 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 31 00:42:56.309430 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:42:56.309361 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Oct 31 00:43:01.773787 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:01.773759 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:43:02.582854 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:02.582811 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 31 00:43:03.499883 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:03.499842 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 31 00:43:04.313654 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:04.313601 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs"] Oct 31 00:43:04.313946 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:04.313893 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" containerID="cri-o://9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc" gracePeriod=30 Oct 31 00:43:04.313946 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:04.313928 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kube-rbac-proxy" containerID="cri-o://971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6" gracePeriod=30 Oct 31 00:43:04.641309 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:04.641220 5821 generic.go:356] "Generic (PLEG): container finished" podID="9fa8f943-8879-473d-a84f-f907645afaf3" containerID="971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6" exitCode=0 Oct 31 00:43:04.641309 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:04.641291 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" event={"ID":"9fa8f943-8879-473d-a84f-f907645afaf3","Type":"ContainerDied","Data":"971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6"} Oct 31 00:43:06.304367 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:06.304328 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused" Oct 31 00:43:09.256487 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.256464 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:43:09.361496 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.361402 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa8f943-8879-473d-a84f-f907645afaf3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fa8f943-8879-473d-a84f-f907645afaf3-kserve-provision-location\") pod \"9fa8f943-8879-473d-a84f-f907645afaf3\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " Oct 31 00:43:09.361496 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.361454 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa8f943-8879-473d-a84f-f907645afaf3-kube-api-access-5q55z\" (UniqueName: \"kubernetes.io/projected/9fa8f943-8879-473d-a84f-f907645afaf3-kube-api-access-5q55z\") pod \"9fa8f943-8879-473d-a84f-f907645afaf3\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " Oct 31 00:43:09.361496 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.361487 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-cabundle-cert\") pod \"9fa8f943-8879-473d-a84f-f907645afaf3\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " Oct 31 00:43:09.361807 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.361540 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-kube-rbac-proxy-sar-config\") pod \"9fa8f943-8879-473d-a84f-f907645afaf3\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " Oct 31 00:43:09.361807 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.361599 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fa8f943-8879-473d-a84f-f907645afaf3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fa8f943-8879-473d-a84f-f907645afaf3-proxy-tls\") pod \"9fa8f943-8879-473d-a84f-f907645afaf3\" (UID: \"9fa8f943-8879-473d-a84f-f907645afaf3\") " Oct 31 00:43:09.361887 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.361765 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa8f943-8879-473d-a84f-f907645afaf3-kserve-provision-location" pod "9fa8f943-8879-473d-a84f-f907645afaf3" (UID: "9fa8f943-8879-473d-a84f-f907645afaf3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:09.361948 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.361899 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-cabundle-cert" pod "9fa8f943-8879-473d-a84f-f907645afaf3" (UID: "9fa8f943-8879-473d-a84f-f907645afaf3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:09.361948 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.361913 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-kube-rbac-proxy-sar-config" pod "9fa8f943-8879-473d-a84f-f907645afaf3" (UID: "9fa8f943-8879-473d-a84f-f907645afaf3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:09.363853 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.363826 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa8f943-8879-473d-a84f-f907645afaf3-proxy-tls" pod "9fa8f943-8879-473d-a84f-f907645afaf3" (UID: "9fa8f943-8879-473d-a84f-f907645afaf3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:09.363915 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.363895 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa8f943-8879-473d-a84f-f907645afaf3-kube-api-access-5q55z" pod "9fa8f943-8879-473d-a84f-f907645afaf3" (UID: "9fa8f943-8879-473d-a84f-f907645afaf3"). InnerVolumeSpecName "kube-api-access-5q55z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:09.462580 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.462531 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:09.462580 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.462574 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fa8f943-8879-473d-a84f-f907645afaf3-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:09.462580 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.462589 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fa8f943-8879-473d-a84f-f907645afaf3-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:09.462580 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.462597 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q55z\" (UniqueName: \"kubernetes.io/projected/9fa8f943-8879-473d-a84f-f907645afaf3-kube-api-access-5q55z\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:09.462853 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.462606 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fa8f943-8879-473d-a84f-f907645afaf3-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:09.659725 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.659643 5821 generic.go:356] "Generic (PLEG): container finished" podID="9fa8f943-8879-473d-a84f-f907645afaf3" containerID="9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc" exitCode=0 Oct 31 00:43:09.659725 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.659692 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" Oct 31 00:43:09.659725 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.659704 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" event={"ID":"9fa8f943-8879-473d-a84f-f907645afaf3","Type":"ContainerDied","Data":"9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc"} Oct 31 00:43:09.659929 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.659756 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs" event={"ID":"9fa8f943-8879-473d-a84f-f907645afaf3","Type":"ContainerDied","Data":"e96d0ee5ce8a68dc8f7c450dc8cf6b3d87ee4181fef9d50508983661268a45ee"} Oct 31 00:43:09.659929 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.659784 5821 scope.go:122] "RemoveContainer" containerID="971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6" Oct 31 00:43:09.668752 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.668733 5821 scope.go:122] "RemoveContainer" containerID="9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc" Oct 31 00:43:09.676329 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.676312 5821 scope.go:122] "RemoveContainer" containerID="3f041f067f77af115e4cbc24ff62aedaec19be72ad5d2ebb21efc40d396edeaa" Oct 31 00:43:09.680910 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.680891 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs"] Oct 31 00:43:09.684348 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.684333 5821 scope.go:122] "RemoveContainer" containerID="971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6" Oct 31 00:43:09.684706 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:43:09.684676 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6\": container with ID starting with 971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6 not found: ID does not exist" containerID="971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6" Oct 31 00:43:09.684818 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.684708 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6"} err="failed to get container status \"971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6\": rpc error: code = NotFound desc = could not find container \"971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6\": container with ID starting with 971a1be980610ce0f42b71850b03c6d8651777d51d07a7f2d8006d52ea1ccbd6 not found: ID does not exist" Oct 31 00:43:09.684818 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.684732 5821 scope.go:122] "RemoveContainer" containerID="9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc" Oct 31 00:43:09.684977 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:43:09.684960 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc\": container with ID starting with 9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc not found: ID does not exist" containerID="9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc" Oct 31 00:43:09.685048 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.684981 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc"} err="failed to get container status \"9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc\": rpc error: code = NotFound desc = could not find container \"9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc\": container with ID starting with 9b8f78e452b666a7bc2667b8ce71b8dc7e6065704ec5f78d7639f2e2b98d99cc not found: ID does not exist" Oct 31 00:43:09.685048 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.684992 5821 scope.go:122] "RemoveContainer" containerID="3f041f067f77af115e4cbc24ff62aedaec19be72ad5d2ebb21efc40d396edeaa" Oct 31 00:43:09.685156 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.685105 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-rollback-predictor-5bb5f8559f-4krqs"] Oct 31 00:43:09.685225 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:43:09.685210 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f041f067f77af115e4cbc24ff62aedaec19be72ad5d2ebb21efc40d396edeaa\": container with ID starting with 3f041f067f77af115e4cbc24ff62aedaec19be72ad5d2ebb21efc40d396edeaa not found: ID does not exist" containerID="3f041f067f77af115e4cbc24ff62aedaec19be72ad5d2ebb21efc40d396edeaa" Oct 31 00:43:09.685264 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:09.685243 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f041f067f77af115e4cbc24ff62aedaec19be72ad5d2ebb21efc40d396edeaa"} err="failed to get container status \"3f041f067f77af115e4cbc24ff62aedaec19be72ad5d2ebb21efc40d396edeaa\": rpc error: code = NotFound desc = could not find container \"3f041f067f77af115e4cbc24ff62aedaec19be72ad5d2ebb21efc40d396edeaa\": container with ID starting with 3f041f067f77af115e4cbc24ff62aedaec19be72ad5d2ebb21efc40d396edeaa not found: ID does not exist" Oct 31 00:43:10.777401 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:10.777370 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" path="/var/lib/kubelet/pods/9fa8f943-8879-473d-a84f-f907645afaf3/volumes" Oct 31 00:43:10.989693 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:10.989661 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr"] Oct 31 00:43:10.989955 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:10.989943 5821 cpu_manager.go:414] "containerMap: removing container" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kube-rbac-proxy" Oct 31 00:43:10.990004 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:10.989957 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kube-rbac-proxy" Oct 31 00:43:10.990004 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:10.989979 5821 cpu_manager.go:414] "containerMap: removing container" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="storage-initializer" Oct 31 00:43:10.990004 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:10.989984 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="storage-initializer" Oct 31 00:43:10.990095 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:10.990012 5821 cpu_manager.go:414] "containerMap: removing container" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" Oct 31 00:43:10.990095 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:10.990017 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" Oct 31 00:43:10.990095 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:10.990077 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kserve-container" Oct 31 00:43:10.990095 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:10.990084 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa8f943-8879-473d-a84f-f907645afaf3" containerName="kube-rbac-proxy" Oct 31 00:43:10.994380 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:10.994358 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.002384 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.002358 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr"] Oct 31 00:43:11.026532 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.026500 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"openshift-service-ca.crt\"" Oct 31 00:43:11.026674 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.026519 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"kube-root-ca.crt\"" Oct 31 00:43:11.026674 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.026500 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\"" Oct 31 00:43:11.026768 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.026753 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:43:11.026807 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.026768 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"isvc-canary-stop-predictor-serving-cert\"" Oct 31 00:43:11.027022 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.027011 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"seaweedfs-s3-creds\"" Oct 31 00:43:11.027055 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.027040 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"default-dockercfg-84b4v\"" Oct 31 00:43:11.073668 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.073566 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct2l6\" (UniqueName: \"kubernetes.io/projected/bef7051c-fa9f-4885-ae93-cff266270020-kube-api-access-ct2l6\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.073668 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.073603 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.073847 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.073729 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-cabundle-cert\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.073847 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.073762 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef7051c-fa9f-4885-ae93-cff266270020-proxy-tls\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.073847 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.073784 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef7051c-fa9f-4885-ae93-cff266270020-kserve-provision-location\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.174632 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.174593 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef7051c-fa9f-4885-ae93-cff266270020-kserve-provision-location\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.174806 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.174678 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct2l6\" (UniqueName: \"kubernetes.io/projected/bef7051c-fa9f-4885-ae93-cff266270020-kube-api-access-ct2l6\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.174806 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.174708 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.174806 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.174748 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-cabundle-cert\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.174806 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.174781 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef7051c-fa9f-4885-ae93-cff266270020-proxy-tls\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.175037 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.175009 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef7051c-fa9f-4885-ae93-cff266270020-kserve-provision-location\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.175365 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.175342 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-cabundle-cert\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.175365 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.175360 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.177307 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.177284 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef7051c-fa9f-4885-ae93-cff266270020-proxy-tls\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.183272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.183254 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct2l6\" (UniqueName: \"kubernetes.io/projected/bef7051c-fa9f-4885-ae93-cff266270020-kube-api-access-ct2l6\") pod \"isvc-canary-stop-predictor-65dd46dbb7-687gr\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.313535 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.313494 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:11.590532 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.590457 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr"] Oct 31 00:43:11.597693 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:43:11.597660 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef7051c_fa9f_4885_ae93_cff266270020.slice/crio-0d7715e9f6d65fbafeb59d0ecdb7c531bc57f0bb5d6f49de4d0a38ce868115ac WatchSource:0}: Error finding container 0d7715e9f6d65fbafeb59d0ecdb7c531bc57f0bb5d6f49de4d0a38ce868115ac: Status 404 returned error can't find the container with id 0d7715e9f6d65fbafeb59d0ecdb7c531bc57f0bb5d6f49de4d0a38ce868115ac Oct 31 00:43:11.670089 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:11.670063 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" event={"ID":"bef7051c-fa9f-4885-ae93-cff266270020","Type":"ContainerStarted","Data":"0d7715e9f6d65fbafeb59d0ecdb7c531bc57f0bb5d6f49de4d0a38ce868115ac"} Oct 31 00:43:12.582516 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:12.582479 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 31 00:43:12.675238 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:12.675198 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" event={"ID":"bef7051c-fa9f-4885-ae93-cff266270020","Type":"ContainerStarted","Data":"cc12223098c8ee0f77824e8de442e5345078348a84732f93bed9f8c03768f9ee"} Oct 31 00:43:13.500651 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:13.500600 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 31 00:43:15.687008 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:15.686975 5821 generic.go:356] "Generic (PLEG): container finished" podID="bef7051c-fa9f-4885-ae93-cff266270020" containerID="cc12223098c8ee0f77824e8de442e5345078348a84732f93bed9f8c03768f9ee" exitCode=0 Oct 31 00:43:15.687538 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:15.687050 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" event={"ID":"bef7051c-fa9f-4885-ae93-cff266270020","Type":"ContainerDied","Data":"cc12223098c8ee0f77824e8de442e5345078348a84732f93bed9f8c03768f9ee"} Oct 31 00:43:16.693689 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:16.693652 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" event={"ID":"bef7051c-fa9f-4885-ae93-cff266270020","Type":"ContainerStarted","Data":"b5175d17647d92c10882faf4f80df08bf701a33cc111838d8b5211993ce13ade"} Oct 31 00:43:16.694201 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:16.693713 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" event={"ID":"bef7051c-fa9f-4885-ae93-cff266270020","Type":"ContainerStarted","Data":"79b4af9c160f770a76c4173922bf1f41ccd4f268918e98abee1dd9fc853ee6d5"} Oct 31 00:43:16.694201 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:16.694045 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:16.694330 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:16.694198 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:16.695819 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:16.695782 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 31 00:43:16.713123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:16.712982 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podStartSLOduration=2.942485592 podStartE2EDuration="6.712967755s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.770482163s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:16.712195032 +0000 UTC m=+780.073777670" watchObservedRunningTime="2026-10-31 00:43:16.712967755 +0000 UTC m=+780.074550394" Oct 31 00:43:17.698309 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:17.698260 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 31 00:43:19.838356 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:43:19.838320 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:43:22.582529 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:22.582492 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 31 00:43:22.703066 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:22.703039 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:43:22.703546 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:22.703520 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 31 00:43:23.500126 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:23.500081 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 31 00:43:32.582506 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:32.582466 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 31 00:43:32.703999 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:32.703964 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 31 00:43:33.500692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:33.500650 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 31 00:43:34.773486 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:34.773434 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 31 00:43:42.583236 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:42.583193 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 31 00:43:42.704106 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:42.704065 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 31 00:43:44.773583 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:44.773542 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 31 00:43:49.838698 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:43:49.838658 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:43:52.583278 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:52.583252 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:43:52.704257 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:52.704211 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 31 00:43:54.777574 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:43:54.777550 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:44:02.704388 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:02.704310 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 31 00:44:12.703523 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:12.703485 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 31 00:44:19.837526 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:44:19.837484 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:44:22.145225 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.145194 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b"] Oct 31 00:44:22.145802 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.145495 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" containerID="cri-o://44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49" gracePeriod=30 Oct 31 00:44:22.145863 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.145841 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kube-rbac-proxy" containerID="cri-o://dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183" gracePeriod=30 Oct 31 00:44:22.177223 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.177195 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx"] Oct 31 00:44:22.177496 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.177458 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" containerID="cri-o://899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9" gracePeriod=30 Oct 31 00:44:22.177587 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.177535 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kube-rbac-proxy" containerID="cri-o://acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b" gracePeriod=30 Oct 31 00:44:22.578211 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.578167 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused" Oct 31 00:44:22.582508 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.582487 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 31 00:44:22.703701 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.703666 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 31 00:44:22.939243 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.939209 5821 generic.go:356] "Generic (PLEG): container finished" podID="39731e33-baa9-4164-881d-9dc637c3281b" containerID="dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183" exitCode=0 Oct 31 00:44:22.939404 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.939282 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" event={"ID":"39731e33-baa9-4164-881d-9dc637c3281b","Type":"ContainerDied","Data":"dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183"} Oct 31 00:44:22.941247 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.941224 5821 generic.go:356] "Generic (PLEG): container finished" podID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerID="acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b" exitCode=0 Oct 31 00:44:22.941369 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:22.941294 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" event={"ID":"960da667-4f1b-45a2-ac83-c664f62aca2c","Type":"ContainerDied","Data":"acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b"} Oct 31 00:44:23.492665 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:23.492604 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused" Oct 31 00:44:24.773900 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:24.773848 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 31 00:44:26.517970 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.517946 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:44:26.606244 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.606136 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-rbac-proxy-sar-config\") pod \"960da667-4f1b-45a2-ac83-c664f62aca2c\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " Oct 31 00:44:26.606244 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.606187 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960da667-4f1b-45a2-ac83-c664f62aca2c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/960da667-4f1b-45a2-ac83-c664f62aca2c-kserve-provision-location\") pod \"960da667-4f1b-45a2-ac83-c664f62aca2c\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " Oct 31 00:44:26.606244 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.606215 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-cabundle-cert\") pod \"960da667-4f1b-45a2-ac83-c664f62aca2c\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " Oct 31 00:44:26.606529 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.606255 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-api-access-tmqml\" (UniqueName: \"kubernetes.io/projected/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-api-access-tmqml\") pod \"960da667-4f1b-45a2-ac83-c664f62aca2c\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " Oct 31 00:44:26.606529 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.606356 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/960da667-4f1b-45a2-ac83-c664f62aca2c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/960da667-4f1b-45a2-ac83-c664f62aca2c-proxy-tls\") pod \"960da667-4f1b-45a2-ac83-c664f62aca2c\" (UID: \"960da667-4f1b-45a2-ac83-c664f62aca2c\") " Oct 31 00:44:26.606529 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.606503 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960da667-4f1b-45a2-ac83-c664f62aca2c-kserve-provision-location" pod "960da667-4f1b-45a2-ac83-c664f62aca2c" (UID: "960da667-4f1b-45a2-ac83-c664f62aca2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:26.606676 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.606551 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-rbac-proxy-sar-config" pod "960da667-4f1b-45a2-ac83-c664f62aca2c" (UID: "960da667-4f1b-45a2-ac83-c664f62aca2c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:26.606676 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.606566 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-cabundle-cert" pod "960da667-4f1b-45a2-ac83-c664f62aca2c" (UID: "960da667-4f1b-45a2-ac83-c664f62aca2c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:26.606676 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.606645 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/960da667-4f1b-45a2-ac83-c664f62aca2c-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:26.608590 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.608567 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/960da667-4f1b-45a2-ac83-c664f62aca2c-proxy-tls" pod "960da667-4f1b-45a2-ac83-c664f62aca2c" (UID: "960da667-4f1b-45a2-ac83-c664f62aca2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:26.608741 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.608717 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-api-access-tmqml" pod "960da667-4f1b-45a2-ac83-c664f62aca2c" (UID: "960da667-4f1b-45a2-ac83-c664f62aca2c"). InnerVolumeSpecName "kube-api-access-tmqml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:26.707857 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.707818 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:26.707857 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.707852 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/960da667-4f1b-45a2-ac83-c664f62aca2c-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:26.707857 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.707864 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmqml\" (UniqueName: \"kubernetes.io/projected/960da667-4f1b-45a2-ac83-c664f62aca2c-kube-api-access-tmqml\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:26.708088 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.707874 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960da667-4f1b-45a2-ac83-c664f62aca2c-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:26.956110 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.956072 5821 generic.go:356] "Generic (PLEG): container finished" podID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerID="899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9" exitCode=0 Oct 31 00:44:26.956263 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.956138 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" Oct 31 00:44:26.956263 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.956138 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" event={"ID":"960da667-4f1b-45a2-ac83-c664f62aca2c","Type":"ContainerDied","Data":"899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9"} Oct 31 00:44:26.956263 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.956200 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx" event={"ID":"960da667-4f1b-45a2-ac83-c664f62aca2c","Type":"ContainerDied","Data":"272f3af34cfc09f0538b3e3aa4dd0a278cab9e6d8348d451fe1be9a754ec69eb"} Oct 31 00:44:26.956263 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.956239 5821 scope.go:122] "RemoveContainer" containerID="acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b" Oct 31 00:44:26.964324 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.964301 5821 scope.go:122] "RemoveContainer" containerID="899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9" Oct 31 00:44:26.971702 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.971685 5821 scope.go:122] "RemoveContainer" containerID="d200e8c4ca40f6134890605a67977ccec0314ed124d1998c289ece4dac5f5372" Oct 31 00:44:26.974180 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.974156 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx"] Oct 31 00:44:26.978271 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.978248 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-ad17a-predictor-555d5c4468-xmxsx"] Oct 31 00:44:26.979826 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.979810 5821 scope.go:122] "RemoveContainer" containerID="acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b" Oct 31 00:44:26.980106 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:44:26.980078 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b\": container with ID starting with acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b not found: ID does not exist" containerID="acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b" Oct 31 00:44:26.980176 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.980117 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b"} err="failed to get container status \"acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b\": rpc error: code = NotFound desc = could not find container \"acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b\": container with ID starting with acece543341535f8ee3424002b8e20606f91399cce6ee4e9bbdb42017c933b1b not found: ID does not exist" Oct 31 00:44:26.980176 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.980143 5821 scope.go:122] "RemoveContainer" containerID="899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9" Oct 31 00:44:26.980441 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:44:26.980424 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9\": container with ID starting with 899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9 not found: ID does not exist" containerID="899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9" Oct 31 00:44:26.980485 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.980448 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9"} err="failed to get container status \"899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9\": rpc error: code = NotFound desc = could not find container \"899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9\": container with ID starting with 899f95eaad3e14182684ed80405c1789e2b2d080dbc352a7248284d74f715ec9 not found: ID does not exist" Oct 31 00:44:26.980485 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.980462 5821 scope.go:122] "RemoveContainer" containerID="d200e8c4ca40f6134890605a67977ccec0314ed124d1998c289ece4dac5f5372" Oct 31 00:44:26.980716 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:44:26.980700 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d200e8c4ca40f6134890605a67977ccec0314ed124d1998c289ece4dac5f5372\": container with ID starting with d200e8c4ca40f6134890605a67977ccec0314ed124d1998c289ece4dac5f5372 not found: ID does not exist" containerID="d200e8c4ca40f6134890605a67977ccec0314ed124d1998c289ece4dac5f5372" Oct 31 00:44:26.980760 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:26.980721 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d200e8c4ca40f6134890605a67977ccec0314ed124d1998c289ece4dac5f5372"} err="failed to get container status \"d200e8c4ca40f6134890605a67977ccec0314ed124d1998c289ece4dac5f5372\": rpc error: code = NotFound desc = could not find container \"d200e8c4ca40f6134890605a67977ccec0314ed124d1998c289ece4dac5f5372\": container with ID starting with d200e8c4ca40f6134890605a67977ccec0314ed124d1998c289ece4dac5f5372 not found: ID does not exist" Oct 31 00:44:27.278270 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.278248 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:44:27.414556 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.414489 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39731e33-baa9-4164-881d-9dc637c3281b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/39731e33-baa9-4164-881d-9dc637c3281b-proxy-tls\") pod \"39731e33-baa9-4164-881d-9dc637c3281b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " Oct 31 00:44:27.414556 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.414523 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-kube-rbac-proxy-sar-config\") pod \"39731e33-baa9-4164-881d-9dc637c3281b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " Oct 31 00:44:27.414790 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.414576 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39731e33-baa9-4164-881d-9dc637c3281b-kube-api-access-rxp9n\" (UniqueName: \"kubernetes.io/projected/39731e33-baa9-4164-881d-9dc637c3281b-kube-api-access-rxp9n\") pod \"39731e33-baa9-4164-881d-9dc637c3281b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " Oct 31 00:44:27.414790 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.414635 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39731e33-baa9-4164-881d-9dc637c3281b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39731e33-baa9-4164-881d-9dc637c3281b-kserve-provision-location\") pod \"39731e33-baa9-4164-881d-9dc637c3281b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " Oct 31 00:44:27.414790 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.414675 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-cabundle-cert\") pod \"39731e33-baa9-4164-881d-9dc637c3281b\" (UID: \"39731e33-baa9-4164-881d-9dc637c3281b\") " Oct 31 00:44:27.415003 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.414969 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-kube-rbac-proxy-sar-config" pod "39731e33-baa9-4164-881d-9dc637c3281b" (UID: "39731e33-baa9-4164-881d-9dc637c3281b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:27.415085 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.414980 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39731e33-baa9-4164-881d-9dc637c3281b-kserve-provision-location" pod "39731e33-baa9-4164-881d-9dc637c3281b" (UID: "39731e33-baa9-4164-881d-9dc637c3281b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:27.415085 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.415042 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-cabundle-cert" pod "39731e33-baa9-4164-881d-9dc637c3281b" (UID: "39731e33-baa9-4164-881d-9dc637c3281b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:27.416787 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.416760 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39731e33-baa9-4164-881d-9dc637c3281b-proxy-tls" pod "39731e33-baa9-4164-881d-9dc637c3281b" (UID: "39731e33-baa9-4164-881d-9dc637c3281b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:27.416876 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.416827 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39731e33-baa9-4164-881d-9dc637c3281b-kube-api-access-rxp9n" pod "39731e33-baa9-4164-881d-9dc637c3281b" (UID: "39731e33-baa9-4164-881d-9dc637c3281b"). InnerVolumeSpecName "kube-api-access-rxp9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:27.515870 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.515833 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39731e33-baa9-4164-881d-9dc637c3281b-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:27.515870 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.515864 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:27.515870 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.515875 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxp9n\" (UniqueName: \"kubernetes.io/projected/39731e33-baa9-4164-881d-9dc637c3281b-kube-api-access-rxp9n\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:27.516105 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.515884 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39731e33-baa9-4164-881d-9dc637c3281b-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:27.516105 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.515895 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39731e33-baa9-4164-881d-9dc637c3281b-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:27.961008 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.960970 5821 generic.go:356] "Generic (PLEG): container finished" podID="39731e33-baa9-4164-881d-9dc637c3281b" containerID="44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49" exitCode=0 Oct 31 00:44:27.961458 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.961043 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" Oct 31 00:44:27.961458 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.961062 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" event={"ID":"39731e33-baa9-4164-881d-9dc637c3281b","Type":"ContainerDied","Data":"44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49"} Oct 31 00:44:27.961458 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.961120 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b" event={"ID":"39731e33-baa9-4164-881d-9dc637c3281b","Type":"ContainerDied","Data":"d1de3fd1eb75eb8c05c82dd196ac070d0e7e8abe12cac057ca101ec2e51ee030"} Oct 31 00:44:27.961458 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.961159 5821 scope.go:122] "RemoveContainer" containerID="dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183" Oct 31 00:44:27.969863 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.969847 5821 scope.go:122] "RemoveContainer" containerID="44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49" Oct 31 00:44:27.976887 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.976872 5821 scope.go:122] "RemoveContainer" containerID="ee815cbcfdb4deff2dfd9359feeb3e83f0f1d969b2eac88c80e67a771ad77553" Oct 31 00:44:27.982409 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.982378 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b"] Oct 31 00:44:27.984336 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.984319 5821 scope.go:122] "RemoveContainer" containerID="dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183" Oct 31 00:44:27.984610 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:44:27.984592 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183\": container with ID starting with dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183 not found: ID does not exist" containerID="dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183" Oct 31 00:44:27.984680 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.984633 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183"} err="failed to get container status \"dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183\": rpc error: code = NotFound desc = could not find container \"dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183\": container with ID starting with dbf6d9f7378570d8a36d10ba0886867537fbfee31488fef452021b41cd651183 not found: ID does not exist" Oct 31 00:44:27.984680 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.984651 5821 scope.go:122] "RemoveContainer" containerID="44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49" Oct 31 00:44:27.984903 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:44:27.984885 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49\": container with ID starting with 44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49 not found: ID does not exist" containerID="44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49" Oct 31 00:44:27.984976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.984907 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49"} err="failed to get container status \"44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49\": rpc error: code = NotFound desc = could not find container \"44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49\": container with ID starting with 44bdd32c4c83d5748820920b8831c8d4a4a07b08ad87693062ea3f31ed146d49 not found: ID does not exist" Oct 31 00:44:27.984976 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.984923 5821 scope.go:122] "RemoveContainer" containerID="ee815cbcfdb4deff2dfd9359feeb3e83f0f1d969b2eac88c80e67a771ad77553" Oct 31 00:44:27.985407 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:44:27.985378 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee815cbcfdb4deff2dfd9359feeb3e83f0f1d969b2eac88c80e67a771ad77553\": container with ID starting with ee815cbcfdb4deff2dfd9359feeb3e83f0f1d969b2eac88c80e67a771ad77553 not found: ID does not exist" containerID="ee815cbcfdb4deff2dfd9359feeb3e83f0f1d969b2eac88c80e67a771ad77553" Oct 31 00:44:27.985496 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.985413 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee815cbcfdb4deff2dfd9359feeb3e83f0f1d969b2eac88c80e67a771ad77553"} err="failed to get container status \"ee815cbcfdb4deff2dfd9359feeb3e83f0f1d969b2eac88c80e67a771ad77553\": rpc error: code = NotFound desc = could not find container \"ee815cbcfdb4deff2dfd9359feeb3e83f0f1d969b2eac88c80e67a771ad77553\": container with ID starting with ee815cbcfdb4deff2dfd9359feeb3e83f0f1d969b2eac88c80e67a771ad77553 not found: ID does not exist" Oct 31 00:44:27.986416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:27.986399 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-ad17a-predictor-58f67d7454-d898b"] Oct 31 00:44:28.773484 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:28.773441 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 31 00:44:28.777179 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:28.777153 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39731e33-baa9-4164-881d-9dc637c3281b" path="/var/lib/kubelet/pods/39731e33-baa9-4164-881d-9dc637c3281b/volumes" Oct 31 00:44:28.777615 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:28.777603 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" path="/var/lib/kubelet/pods/960da667-4f1b-45a2-ac83-c664f62aca2c/volumes" Oct 31 00:44:29.588737 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.588703 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw"] Oct 31 00:44:29.589154 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589134 5821 cpu_manager.go:414] "containerMap: removing container" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="storage-initializer" Oct 31 00:44:29.589232 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589159 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="storage-initializer" Oct 31 00:44:29.589232 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589177 5821 cpu_manager.go:414] "containerMap: removing container" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="storage-initializer" Oct 31 00:44:29.589232 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589186 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="storage-initializer" Oct 31 00:44:29.589232 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589198 5821 cpu_manager.go:414] "containerMap: removing container" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kube-rbac-proxy" Oct 31 00:44:29.589232 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589205 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kube-rbac-proxy" Oct 31 00:44:29.589232 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589214 5821 cpu_manager.go:414] "containerMap: removing container" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kube-rbac-proxy" Oct 31 00:44:29.589232 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589222 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kube-rbac-proxy" Oct 31 00:44:29.589510 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589235 5821 cpu_manager.go:414] "containerMap: removing container" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" Oct 31 00:44:29.589510 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589242 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" Oct 31 00:44:29.589510 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589285 5821 cpu_manager.go:414] "containerMap: removing container" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" Oct 31 00:44:29.589510 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589293 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" Oct 31 00:44:29.589510 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589360 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kserve-container" Oct 31 00:44:29.589510 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589370 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="39731e33-baa9-4164-881d-9dc637c3281b" containerName="kube-rbac-proxy" Oct 31 00:44:29.589510 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589376 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kube-rbac-proxy" Oct 31 00:44:29.589510 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.589383 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="960da667-4f1b-45a2-ac83-c664f62aca2c" containerName="kserve-container" Oct 31 00:44:29.594107 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.594089 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.601088 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.600759 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"openshift-service-ca.crt\"" Oct 31 00:44:29.601088 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.600942 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:44:29.601088 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.600940 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"seaweedfs-s3-creds\"" Oct 31 00:44:29.601088 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.600990 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-sklearn-graph-raw-hpa-d7985-predictor-serving-cert\"" Oct 31 00:44:29.601088 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.601040 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"default-dockercfg-gs5fn\"" Oct 31 00:44:29.601088 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.601057 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-sklearn-graph-raw-hpa-d7985-kube-rbac-proxy-sar-config\"" Oct 31 00:44:29.601459 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.601160 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"kube-root-ca.crt\"" Oct 31 00:44:29.602039 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.602017 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw"] Oct 31 00:44:29.732005 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.731934 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72kg\" (UniqueName: \"kubernetes.io/projected/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-api-access-v72kg\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.732005 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.731987 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.732338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.732052 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.732338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.732260 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ffdc475-4866-4af3-be5a-8e406b84459b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.732338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.732333 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ffdc475-4866-4af3-be5a-8e406b84459b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.801994 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.801961 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq"] Oct 31 00:44:29.805323 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.805306 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:29.809046 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.809021 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-xgboost-graph-raw-hpa-d7985-kube-rbac-proxy-sar-config\"" Oct 31 00:44:29.809164 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.809064 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-xgboost-graph-raw-hpa-d7985-predictor-serving-cert\"" Oct 31 00:44:29.815549 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.815528 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq"] Oct 31 00:44:29.833162 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.833131 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72kg\" (UniqueName: \"kubernetes.io/projected/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-api-access-v72kg\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.833162 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.833164 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.833348 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.833198 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.833348 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.833219 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ffdc475-4866-4af3-be5a-8e406b84459b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.833348 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.833261 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ffdc475-4866-4af3-be5a-8e406b84459b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.833715 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.833693 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ffdc475-4866-4af3-be5a-8e406b84459b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.833859 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.833841 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.833916 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.833902 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.835849 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.835830 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ffdc475-4866-4af3-be5a-8e406b84459b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.841477 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.841428 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72kg\" (UniqueName: \"kubernetes.io/projected/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-api-access-v72kg\") pod \"isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.913418 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.913390 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:29.934505 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.934470 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f51798a-0809-4429-b5f1-90d20751d38c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:29.934644 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.934546 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:29.934644 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.934579 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:29.934787 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.934664 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m6zq\" (UniqueName: \"kubernetes.io/projected/9f51798a-0809-4429-b5f1-90d20751d38c-kube-api-access-2m6zq\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:29.934787 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:29.934715 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f51798a-0809-4429-b5f1-90d20751d38c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:30.035916 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.035876 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f51798a-0809-4429-b5f1-90d20751d38c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:30.035916 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.035919 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:30.036157 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.035945 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:30.036157 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.035974 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m6zq\" (UniqueName: \"kubernetes.io/projected/9f51798a-0809-4429-b5f1-90d20751d38c-kube-api-access-2m6zq\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:30.036157 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.036016 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f51798a-0809-4429-b5f1-90d20751d38c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:30.036465 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.036443 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f51798a-0809-4429-b5f1-90d20751d38c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:30.036788 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.036764 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:30.036837 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.036764 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:30.038520 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.038498 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f51798a-0809-4429-b5f1-90d20751d38c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:30.045489 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.045467 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m6zq\" (UniqueName: \"kubernetes.io/projected/9f51798a-0809-4429-b5f1-90d20751d38c-kube-api-access-2m6zq\") pod \"isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:30.123084 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.123026 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:30.186767 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.186738 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw"] Oct 31 00:44:30.189961 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:44:30.189933 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffdc475_4866_4af3_be5a_8e406b84459b.slice/crio-a5dd2d4b396047fe671a46f2fa5e9182fde3549868a2c1274b18769290d7af6d WatchSource:0}: Error finding container a5dd2d4b396047fe671a46f2fa5e9182fde3549868a2c1274b18769290d7af6d: Status 404 returned error can't find the container with id a5dd2d4b396047fe671a46f2fa5e9182fde3549868a2c1274b18769290d7af6d Oct 31 00:44:30.396459 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.396364 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq"] Oct 31 00:44:30.399713 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:44:30.399683 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f51798a_0809_4429_b5f1_90d20751d38c.slice/crio-f7df3e6f25c3a591b109e046207336393c099748b11428aaa269dc3d9baf620f WatchSource:0}: Error finding container f7df3e6f25c3a591b109e046207336393c099748b11428aaa269dc3d9baf620f: Status 404 returned error can't find the container with id f7df3e6f25c3a591b109e046207336393c099748b11428aaa269dc3d9baf620f Oct 31 00:44:30.978383 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.978345 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" event={"ID":"9f51798a-0809-4429-b5f1-90d20751d38c","Type":"ContainerStarted","Data":"35659415b04c5dc86b0ef7a85a8c990a922fcec2b04c2358ede6ffa877898500"} Oct 31 00:44:30.978872 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.978418 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" event={"ID":"9f51798a-0809-4429-b5f1-90d20751d38c","Type":"ContainerStarted","Data":"f7df3e6f25c3a591b109e046207336393c099748b11428aaa269dc3d9baf620f"} Oct 31 00:44:30.980075 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.980048 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" event={"ID":"5ffdc475-4866-4af3-be5a-8e406b84459b","Type":"ContainerStarted","Data":"3e5462ea038bddacb2a8a6cbe561db2ad0edf6e1f8d9e266a249c51ff8ee3d2e"} Oct 31 00:44:30.980164 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:30.980099 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" event={"ID":"5ffdc475-4866-4af3-be5a-8e406b84459b","Type":"ContainerStarted","Data":"a5dd2d4b396047fe671a46f2fa5e9182fde3549868a2c1274b18769290d7af6d"} Oct 31 00:44:34.997042 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:34.997008 5821 generic.go:356] "Generic (PLEG): container finished" podID="9f51798a-0809-4429-b5f1-90d20751d38c" containerID="35659415b04c5dc86b0ef7a85a8c990a922fcec2b04c2358ede6ffa877898500" exitCode=0 Oct 31 00:44:34.997576 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:34.997085 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" event={"ID":"9f51798a-0809-4429-b5f1-90d20751d38c","Type":"ContainerDied","Data":"35659415b04c5dc86b0ef7a85a8c990a922fcec2b04c2358ede6ffa877898500"} Oct 31 00:44:34.998608 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:34.998588 5821 generic.go:356] "Generic (PLEG): container finished" podID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerID="3e5462ea038bddacb2a8a6cbe561db2ad0edf6e1f8d9e266a249c51ff8ee3d2e" exitCode=0 Oct 31 00:44:34.998702 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:34.998655 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" event={"ID":"5ffdc475-4866-4af3-be5a-8e406b84459b","Type":"ContainerDied","Data":"3e5462ea038bddacb2a8a6cbe561db2ad0edf6e1f8d9e266a249c51ff8ee3d2e"} Oct 31 00:44:36.005240 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:36.005205 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" event={"ID":"9f51798a-0809-4429-b5f1-90d20751d38c","Type":"ContainerStarted","Data":"ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d"} Oct 31 00:44:36.005703 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:36.005284 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" event={"ID":"9f51798a-0809-4429-b5f1-90d20751d38c","Type":"ContainerStarted","Data":"dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7"} Oct 31 00:44:36.005703 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:36.005615 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:36.005703 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:36.005683 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:36.007368 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:36.007343 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 31 00:44:36.007530 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:36.007511 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" event={"ID":"5ffdc475-4866-4af3-be5a-8e406b84459b","Type":"ContainerStarted","Data":"8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac"} Oct 31 00:44:36.007582 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:36.007560 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" event={"ID":"5ffdc475-4866-4af3-be5a-8e406b84459b","Type":"ContainerStarted","Data":"a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435"} Oct 31 00:44:36.007772 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:36.007757 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:36.025580 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:36.025540 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podStartSLOduration=3.274102118 podStartE2EDuration="7.0255282s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.751426082s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:36.02405796 +0000 UTC m=+859.385640610" watchObservedRunningTime="2026-10-31 00:44:36.0255282 +0000 UTC m=+859.387110836" Oct 31 00:44:36.042633 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:36.042576 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podStartSLOduration=3.23343254 podStartE2EDuration="7.04256373s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.80913119s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:36.041088102 +0000 UTC m=+859.402670740" watchObservedRunningTime="2026-10-31 00:44:36.04256373 +0000 UTC m=+859.404146366" Oct 31 00:44:37.011425 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:37.011383 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 31 00:44:37.011957 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:37.011540 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:37.012828 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:37.012804 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 31 00:44:38.014448 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:38.014404 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 31 00:44:38.777257 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:38.777231 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:44:39.012811 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:39.012761 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr"] Oct 31 00:44:39.018640 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:39.018571 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" containerID="cri-o://79b4af9c160f770a76c4173922bf1f41ccd4f268918e98abee1dd9fc853ee6d5" gracePeriod=30 Oct 31 00:44:39.018992 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:39.018771 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kube-rbac-proxy" containerID="cri-o://b5175d17647d92c10882faf4f80df08bf701a33cc111838d8b5211993ce13ade" gracePeriod=30 Oct 31 00:44:40.024163 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:40.024130 5821 generic.go:356] "Generic (PLEG): container finished" podID="bef7051c-fa9f-4885-ae93-cff266270020" containerID="b5175d17647d92c10882faf4f80df08bf701a33cc111838d8b5211993ce13ade" exitCode=0 Oct 31 00:44:40.024610 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:40.024206 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" event={"ID":"bef7051c-fa9f-4885-ae93-cff266270020","Type":"ContainerDied","Data":"b5175d17647d92c10882faf4f80df08bf701a33cc111838d8b5211993ce13ade"} Oct 31 00:44:42.016243 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:42.016213 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:44:42.016726 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:42.016685 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 31 00:44:42.698485 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:42.698433 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" Oct 31 00:44:43.019300 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:43.019221 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:44:43.019709 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:43.019687 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 31 00:44:44.041312 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.041280 5821 generic.go:356] "Generic (PLEG): container finished" podID="bef7051c-fa9f-4885-ae93-cff266270020" containerID="79b4af9c160f770a76c4173922bf1f41ccd4f268918e98abee1dd9fc853ee6d5" exitCode=0 Oct 31 00:44:44.041703 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.041348 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" event={"ID":"bef7051c-fa9f-4885-ae93-cff266270020","Type":"ContainerDied","Data":"79b4af9c160f770a76c4173922bf1f41ccd4f268918e98abee1dd9fc853ee6d5"} Oct 31 00:44:44.064503 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.064479 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:44:44.159808 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.159723 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef7051c-fa9f-4885-ae93-cff266270020-kube-api-access-ct2l6\" (UniqueName: \"kubernetes.io/projected/bef7051c-fa9f-4885-ae93-cff266270020-kube-api-access-ct2l6\") pod \"bef7051c-fa9f-4885-ae93-cff266270020\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " Oct 31 00:44:44.159808 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.159758 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-cabundle-cert\") pod \"bef7051c-fa9f-4885-ae93-cff266270020\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " Oct 31 00:44:44.159808 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.159782 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef7051c-fa9f-4885-ae93-cff266270020-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef7051c-fa9f-4885-ae93-cff266270020-kserve-provision-location\") pod \"bef7051c-fa9f-4885-ae93-cff266270020\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " Oct 31 00:44:44.159808 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.159799 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef7051c-fa9f-4885-ae93-cff266270020-proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef7051c-fa9f-4885-ae93-cff266270020-proxy-tls\") pod \"bef7051c-fa9f-4885-ae93-cff266270020\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " Oct 31 00:44:44.160123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.159833 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-kube-rbac-proxy-sar-config\") pod \"bef7051c-fa9f-4885-ae93-cff266270020\" (UID: \"bef7051c-fa9f-4885-ae93-cff266270020\") " Oct 31 00:44:44.160185 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.160117 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef7051c-fa9f-4885-ae93-cff266270020-kserve-provision-location" pod "bef7051c-fa9f-4885-ae93-cff266270020" (UID: "bef7051c-fa9f-4885-ae93-cff266270020"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:44.160185 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.160135 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-cabundle-cert" pod "bef7051c-fa9f-4885-ae93-cff266270020" (UID: "bef7051c-fa9f-4885-ae93-cff266270020"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:44.160345 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.160322 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-kube-rbac-proxy-sar-config" pod "bef7051c-fa9f-4885-ae93-cff266270020" (UID: "bef7051c-fa9f-4885-ae93-cff266270020"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:44.162044 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.162024 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef7051c-fa9f-4885-ae93-cff266270020-kube-api-access-ct2l6" pod "bef7051c-fa9f-4885-ae93-cff266270020" (UID: "bef7051c-fa9f-4885-ae93-cff266270020"). InnerVolumeSpecName "kube-api-access-ct2l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:44.162104 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.162047 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef7051c-fa9f-4885-ae93-cff266270020-proxy-tls" pod "bef7051c-fa9f-4885-ae93-cff266270020" (UID: "bef7051c-fa9f-4885-ae93-cff266270020"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:44.260996 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.260958 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct2l6\" (UniqueName: \"kubernetes.io/projected/bef7051c-fa9f-4885-ae93-cff266270020-kube-api-access-ct2l6\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:44.260996 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.260992 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:44.261207 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.261006 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef7051c-fa9f-4885-ae93-cff266270020-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:44.261207 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.261022 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef7051c-fa9f-4885-ae93-cff266270020-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:44.261207 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:44.261035 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef7051c-fa9f-4885-ae93-cff266270020-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:45.047317 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:45.047242 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" Oct 31 00:44:45.047741 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:45.047243 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr" event={"ID":"bef7051c-fa9f-4885-ae93-cff266270020","Type":"ContainerDied","Data":"0d7715e9f6d65fbafeb59d0ecdb7c531bc57f0bb5d6f49de4d0a38ce868115ac"} Oct 31 00:44:45.047741 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:45.047390 5821 scope.go:122] "RemoveContainer" containerID="b5175d17647d92c10882faf4f80df08bf701a33cc111838d8b5211993ce13ade" Oct 31 00:44:45.055694 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:45.055672 5821 scope.go:122] "RemoveContainer" containerID="79b4af9c160f770a76c4173922bf1f41ccd4f268918e98abee1dd9fc853ee6d5" Oct 31 00:44:45.063066 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:45.063048 5821 scope.go:122] "RemoveContainer" containerID="cc12223098c8ee0f77824e8de442e5345078348a84732f93bed9f8c03768f9ee" Oct 31 00:44:45.066868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:45.065101 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr"] Oct 31 00:44:45.072305 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:45.071449 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-stop-predictor-65dd46dbb7-687gr"] Oct 31 00:44:46.777905 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:46.777871 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef7051c-fa9f-4885-ae93-cff266270020" path="/var/lib/kubelet/pods/bef7051c-fa9f-4885-ae93-cff266270020/volumes" Oct 31 00:44:49.838662 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:44:49.838608 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:44:52.017357 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:52.017317 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 31 00:44:53.019923 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:44:53.019885 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 31 00:45:02.017487 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:45:02.017445 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 31 00:45:03.020147 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:45:03.020107 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 31 00:45:12.016964 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:45:12.016925 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 31 00:45:13.020234 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:45:13.020195 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 31 00:45:19.839358 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:45:19.839308 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:45:22.017077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:45:22.017032 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 31 00:45:23.019995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:45:23.019957 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 31 00:45:32.017170 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:45:32.017089 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 31 00:45:33.019673 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:45:33.019638 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 31 00:45:42.017786 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:45:42.017758 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:45:43.019774 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:45:43.019735 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 31 00:45:43.773794 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:45:43.773753 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 31 00:45:49.838043 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:45:49.837997 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:45:53.774726 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:45:53.774696 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:46:19.838069 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:46:19.838032 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:46:20.104811 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:20.104732 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw"] Oct 31 00:46:20.105027 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:20.104978 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" containerID="cri-o://a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435" gracePeriod=30 Oct 31 00:46:20.105107 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:20.105048 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kube-rbac-proxy" containerID="cri-o://8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac" gracePeriod=30 Oct 31 00:46:20.143537 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:20.143508 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq"] Oct 31 00:46:20.143813 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:20.143755 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" containerID="cri-o://dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7" gracePeriod=30 Oct 31 00:46:20.143813 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:20.143778 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kube-rbac-proxy" containerID="cri-o://ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d" gracePeriod=30 Oct 31 00:46:20.395371 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:20.395276 5821 generic.go:356] "Generic (PLEG): container finished" podID="9f51798a-0809-4429-b5f1-90d20751d38c" containerID="ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d" exitCode=0 Oct 31 00:46:20.395371 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:20.395348 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" event={"ID":"9f51798a-0809-4429-b5f1-90d20751d38c","Type":"ContainerDied","Data":"ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d"} Oct 31 00:46:20.397857 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:20.397827 5821 generic.go:356] "Generic (PLEG): container finished" podID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerID="8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac" exitCode=0 Oct 31 00:46:20.398017 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:20.397903 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" event={"ID":"5ffdc475-4866-4af3-be5a-8e406b84459b","Type":"ContainerDied","Data":"8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac"} Oct 31 00:46:22.011520 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:22.011483 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused" Oct 31 00:46:22.016877 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:22.016838 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 31 00:46:23.015069 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:23.015025 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused" Oct 31 00:46:23.774733 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:23.774694 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 31 00:46:24.185472 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.185442 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:46:24.246257 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.246232 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f51798a-0809-4429-b5f1-90d20751d38c-kube-api-access-2m6zq\" (UniqueName: \"kubernetes.io/projected/9f51798a-0809-4429-b5f1-90d20751d38c-kube-api-access-2m6zq\") pod \"9f51798a-0809-4429-b5f1-90d20751d38c\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " Oct 31 00:46:24.246405 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.246282 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-cabundle-cert\") pod \"9f51798a-0809-4429-b5f1-90d20751d38c\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " Oct 31 00:46:24.246405 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.246307 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f51798a-0809-4429-b5f1-90d20751d38c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f51798a-0809-4429-b5f1-90d20751d38c-kserve-provision-location\") pod \"9f51798a-0809-4429-b5f1-90d20751d38c\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " Oct 31 00:46:24.246405 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.246355 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-kube-rbac-proxy-sar-config\") pod \"9f51798a-0809-4429-b5f1-90d20751d38c\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " Oct 31 00:46:24.246527 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.246415 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f51798a-0809-4429-b5f1-90d20751d38c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f51798a-0809-4429-b5f1-90d20751d38c-proxy-tls\") pod \"9f51798a-0809-4429-b5f1-90d20751d38c\" (UID: \"9f51798a-0809-4429-b5f1-90d20751d38c\") " Oct 31 00:46:24.246700 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.246673 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-cabundle-cert" pod "9f51798a-0809-4429-b5f1-90d20751d38c" (UID: "9f51798a-0809-4429-b5f1-90d20751d38c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:24.246700 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.246684 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f51798a-0809-4429-b5f1-90d20751d38c-kserve-provision-location" pod "9f51798a-0809-4429-b5f1-90d20751d38c" (UID: "9f51798a-0809-4429-b5f1-90d20751d38c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:24.246822 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.246725 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-kube-rbac-proxy-sar-config" pod "9f51798a-0809-4429-b5f1-90d20751d38c" (UID: "9f51798a-0809-4429-b5f1-90d20751d38c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:24.248545 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.248521 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f51798a-0809-4429-b5f1-90d20751d38c-proxy-tls" pod "9f51798a-0809-4429-b5f1-90d20751d38c" (UID: "9f51798a-0809-4429-b5f1-90d20751d38c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:24.248608 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.248570 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f51798a-0809-4429-b5f1-90d20751d38c-kube-api-access-2m6zq" pod "9f51798a-0809-4429-b5f1-90d20751d38c" (UID: "9f51798a-0809-4429-b5f1-90d20751d38c"). InnerVolumeSpecName "kube-api-access-2m6zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:24.347270 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.347176 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:24.347270 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.347222 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f51798a-0809-4429-b5f1-90d20751d38c-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:24.347270 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.347236 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m6zq\" (UniqueName: \"kubernetes.io/projected/9f51798a-0809-4429-b5f1-90d20751d38c-kube-api-access-2m6zq\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:24.347270 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.347248 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f51798a-0809-4429-b5f1-90d20751d38c-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:24.347270 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.347261 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f51798a-0809-4429-b5f1-90d20751d38c-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:24.414550 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.414518 5821 generic.go:356] "Generic (PLEG): container finished" podID="9f51798a-0809-4429-b5f1-90d20751d38c" containerID="dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7" exitCode=0 Oct 31 00:46:24.414705 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.414578 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" Oct 31 00:46:24.414705 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.414606 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" event={"ID":"9f51798a-0809-4429-b5f1-90d20751d38c","Type":"ContainerDied","Data":"dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7"} Oct 31 00:46:24.414705 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.414687 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq" event={"ID":"9f51798a-0809-4429-b5f1-90d20751d38c","Type":"ContainerDied","Data":"f7df3e6f25c3a591b109e046207336393c099748b11428aaa269dc3d9baf620f"} Oct 31 00:46:24.414816 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.414718 5821 scope.go:122] "RemoveContainer" containerID="ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d" Oct 31 00:46:24.422984 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.422970 5821 scope.go:122] "RemoveContainer" containerID="dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7" Oct 31 00:46:24.430518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.430493 5821 scope.go:122] "RemoveContainer" containerID="35659415b04c5dc86b0ef7a85a8c990a922fcec2b04c2358ede6ffa877898500" Oct 31 00:46:24.435682 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.435661 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq"] Oct 31 00:46:24.440345 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.439527 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-xgboost-graph-raw-hpa-d7985-predictor-867f8887c6-sg6nq"] Oct 31 00:46:24.441365 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.441344 5821 scope.go:122] "RemoveContainer" containerID="ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d" Oct 31 00:46:24.441653 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:46:24.441635 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d\": container with ID starting with ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d not found: ID does not exist" containerID="ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d" Oct 31 00:46:24.441711 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.441661 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d"} err="failed to get container status \"ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d\": rpc error: code = NotFound desc = could not find container \"ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d\": container with ID starting with ab762fb422b0e567827b3bf8d21c095feed7c423481e5a7603529e1fcc33b82d not found: ID does not exist" Oct 31 00:46:24.441711 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.441677 5821 scope.go:122] "RemoveContainer" containerID="dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7" Oct 31 00:46:24.441967 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:46:24.441947 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7\": container with ID starting with dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7 not found: ID does not exist" containerID="dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7" Oct 31 00:46:24.442019 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.441976 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7"} err="failed to get container status \"dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7\": rpc error: code = NotFound desc = could not find container \"dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7\": container with ID starting with dbd948f9bb45ad4f578b3d8442ed3aa9065727f89e0039c67f9e8ad19d15a5f7 not found: ID does not exist" Oct 31 00:46:24.442019 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.441991 5821 scope.go:122] "RemoveContainer" containerID="35659415b04c5dc86b0ef7a85a8c990a922fcec2b04c2358ede6ffa877898500" Oct 31 00:46:24.442247 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:46:24.442229 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35659415b04c5dc86b0ef7a85a8c990a922fcec2b04c2358ede6ffa877898500\": container with ID starting with 35659415b04c5dc86b0ef7a85a8c990a922fcec2b04c2358ede6ffa877898500 not found: ID does not exist" containerID="35659415b04c5dc86b0ef7a85a8c990a922fcec2b04c2358ede6ffa877898500" Oct 31 00:46:24.442294 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.442253 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35659415b04c5dc86b0ef7a85a8c990a922fcec2b04c2358ede6ffa877898500"} err="failed to get container status \"35659415b04c5dc86b0ef7a85a8c990a922fcec2b04c2358ede6ffa877898500\": rpc error: code = NotFound desc = could not find container \"35659415b04c5dc86b0ef7a85a8c990a922fcec2b04c2358ede6ffa877898500\": container with ID starting with 35659415b04c5dc86b0ef7a85a8c990a922fcec2b04c2358ede6ffa877898500 not found: ID does not exist" Oct 31 00:46:24.777753 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.777723 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" path="/var/lib/kubelet/pods/9f51798a-0809-4429-b5f1-90d20751d38c/volumes" Oct 31 00:46:24.931762 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:24.931735 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:46:25.051297 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.051236 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-api-access-v72kg\" (UniqueName: \"kubernetes.io/projected/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-api-access-v72kg\") pod \"5ffdc475-4866-4af3-be5a-8e406b84459b\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " Oct 31 00:46:25.051297 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.051269 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-cabundle-cert\") pod \"5ffdc475-4866-4af3-be5a-8e406b84459b\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " Oct 31 00:46:25.051297 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.051289 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ffdc475-4866-4af3-be5a-8e406b84459b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ffdc475-4866-4af3-be5a-8e406b84459b-kserve-provision-location\") pod \"5ffdc475-4866-4af3-be5a-8e406b84459b\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " Oct 31 00:46:25.051526 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.051323 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-rbac-proxy-sar-config\") pod \"5ffdc475-4866-4af3-be5a-8e406b84459b\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " Oct 31 00:46:25.051526 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.051444 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ffdc475-4866-4af3-be5a-8e406b84459b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ffdc475-4866-4af3-be5a-8e406b84459b-proxy-tls\") pod \"5ffdc475-4866-4af3-be5a-8e406b84459b\" (UID: \"5ffdc475-4866-4af3-be5a-8e406b84459b\") " Oct 31 00:46:25.051669 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.051612 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-cabundle-cert" pod "5ffdc475-4866-4af3-be5a-8e406b84459b" (UID: "5ffdc475-4866-4af3-be5a-8e406b84459b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:25.051669 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.051611 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffdc475-4866-4af3-be5a-8e406b84459b-kserve-provision-location" pod "5ffdc475-4866-4af3-be5a-8e406b84459b" (UID: "5ffdc475-4866-4af3-be5a-8e406b84459b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:25.051796 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.051771 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-rbac-proxy-sar-config" pod "5ffdc475-4866-4af3-be5a-8e406b84459b" (UID: "5ffdc475-4866-4af3-be5a-8e406b84459b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:25.053481 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.053453 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-api-access-v72kg" pod "5ffdc475-4866-4af3-be5a-8e406b84459b" (UID: "5ffdc475-4866-4af3-be5a-8e406b84459b"). InnerVolumeSpecName "kube-api-access-v72kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:25.053581 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.053504 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ffdc475-4866-4af3-be5a-8e406b84459b-proxy-tls" pod "5ffdc475-4866-4af3-be5a-8e406b84459b" (UID: "5ffdc475-4866-4af3-be5a-8e406b84459b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:25.152154 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.152127 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ffdc475-4866-4af3-be5a-8e406b84459b-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:25.152154 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.152150 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72kg\" (UniqueName: \"kubernetes.io/projected/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-api-access-v72kg\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:25.152329 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.152160 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:25.152329 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.152170 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ffdc475-4866-4af3-be5a-8e406b84459b-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:25.152329 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.152180 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ffdc475-4866-4af3-be5a-8e406b84459b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:25.421988 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.421951 5821 generic.go:356] "Generic (PLEG): container finished" podID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerID="a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435" exitCode=0 Oct 31 00:46:25.422394 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.422031 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" Oct 31 00:46:25.422394 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.422040 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" event={"ID":"5ffdc475-4866-4af3-be5a-8e406b84459b","Type":"ContainerDied","Data":"a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435"} Oct 31 00:46:25.422394 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.422099 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw" event={"ID":"5ffdc475-4866-4af3-be5a-8e406b84459b","Type":"ContainerDied","Data":"a5dd2d4b396047fe671a46f2fa5e9182fde3549868a2c1274b18769290d7af6d"} Oct 31 00:46:25.422394 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.422127 5821 scope.go:122] "RemoveContainer" containerID="8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac" Oct 31 00:46:25.431028 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.431011 5821 scope.go:122] "RemoveContainer" containerID="a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435" Oct 31 00:46:25.438158 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.438143 5821 scope.go:122] "RemoveContainer" containerID="3e5462ea038bddacb2a8a6cbe561db2ad0edf6e1f8d9e266a249c51ff8ee3d2e" Oct 31 00:46:25.444192 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.444175 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw"] Oct 31 00:46:25.445629 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.445594 5821 scope.go:122] "RemoveContainer" containerID="8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac" Oct 31 00:46:25.445911 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:46:25.445893 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac\": container with ID starting with 8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac not found: ID does not exist" containerID="8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac" Oct 31 00:46:25.445959 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.445920 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac"} err="failed to get container status \"8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac\": rpc error: code = NotFound desc = could not find container \"8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac\": container with ID starting with 8ec64a5169134a51989066cec4c45269bbbe475ea662f71ee5bac23339618dac not found: ID does not exist" Oct 31 00:46:25.445959 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.445936 5821 scope.go:122] "RemoveContainer" containerID="a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435" Oct 31 00:46:25.446166 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:46:25.446145 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435\": container with ID starting with a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435 not found: ID does not exist" containerID="a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435" Oct 31 00:46:25.446206 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.446172 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435"} err="failed to get container status \"a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435\": rpc error: code = NotFound desc = could not find container \"a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435\": container with ID starting with a745eb667f61b90f860fc52fa7de736fa38a3040f22ad604c37f6826837d0435 not found: ID does not exist" Oct 31 00:46:25.446206 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.446189 5821 scope.go:122] "RemoveContainer" containerID="3e5462ea038bddacb2a8a6cbe561db2ad0edf6e1f8d9e266a249c51ff8ee3d2e" Oct 31 00:46:25.446386 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:46:25.446373 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5462ea038bddacb2a8a6cbe561db2ad0edf6e1f8d9e266a249c51ff8ee3d2e\": container with ID starting with 3e5462ea038bddacb2a8a6cbe561db2ad0edf6e1f8d9e266a249c51ff8ee3d2e not found: ID does not exist" containerID="3e5462ea038bddacb2a8a6cbe561db2ad0edf6e1f8d9e266a249c51ff8ee3d2e" Oct 31 00:46:25.446421 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.446389 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5462ea038bddacb2a8a6cbe561db2ad0edf6e1f8d9e266a249c51ff8ee3d2e"} err="failed to get container status \"3e5462ea038bddacb2a8a6cbe561db2ad0edf6e1f8d9e266a249c51ff8ee3d2e\": rpc error: code = NotFound desc = could not find container \"3e5462ea038bddacb2a8a6cbe561db2ad0edf6e1f8d9e266a249c51ff8ee3d2e\": container with ID starting with 3e5462ea038bddacb2a8a6cbe561db2ad0edf6e1f8d9e266a249c51ff8ee3d2e not found: ID does not exist" Oct 31 00:46:25.448552 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:25.448534 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-graph-raw-hpa-d7985-predictor-5f484dc64b-ktkrw"] Oct 31 00:46:26.777416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:26.777375 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" path="/var/lib/kubelet/pods/5ffdc475-4866-4af3-be5a-8e406b84459b/volumes" Oct 31 00:46:27.474156 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474110 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh"] Oct 31 00:46:27.474723 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474704 5821 cpu_manager.go:414] "containerMap: removing container" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kube-rbac-proxy" Oct 31 00:46:27.474840 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474725 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kube-rbac-proxy" Oct 31 00:46:27.474840 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474751 5821 cpu_manager.go:414] "containerMap: removing container" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" Oct 31 00:46:27.474840 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474759 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" Oct 31 00:46:27.474840 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474769 5821 cpu_manager.go:414] "containerMap: removing container" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="storage-initializer" Oct 31 00:46:27.474840 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474777 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="storage-initializer" Oct 31 00:46:27.474840 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474792 5821 cpu_manager.go:414] "containerMap: removing container" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="storage-initializer" Oct 31 00:46:27.474840 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474799 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="storage-initializer" Oct 31 00:46:27.474840 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474816 5821 cpu_manager.go:414] "containerMap: removing container" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kube-rbac-proxy" Oct 31 00:46:27.474840 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474823 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kube-rbac-proxy" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474856 5821 cpu_manager.go:414] "containerMap: removing container" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="storage-initializer" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474863 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="storage-initializer" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474877 5821 cpu_manager.go:414] "containerMap: removing container" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474889 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474916 5821 cpu_manager.go:414] "containerMap: removing container" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474923 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474935 5821 cpu_manager.go:414] "containerMap: removing container" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kube-rbac-proxy" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.474942 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kube-rbac-proxy" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.475059 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kube-rbac-proxy" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.475070 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kserve-container" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.475086 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f51798a-0809-4429-b5f1-90d20751d38c" containerName="kube-rbac-proxy" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.475097 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kserve-container" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.475112 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef7051c-fa9f-4885-ae93-cff266270020" containerName="kserve-container" Oct 31 00:46:27.475208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.475119 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ffdc475-4866-4af3-be5a-8e406b84459b" containerName="kube-rbac-proxy" Oct 31 00:46:27.480636 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.480606 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:27.484886 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.484863 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh"] Oct 31 00:46:27.487888 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.487866 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"message-dumper-raw-233d7-kube-rbac-proxy-sar-config\"" Oct 31 00:46:27.488333 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.487875 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"openshift-service-ca.crt\"" Oct 31 00:46:27.488443 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.487866 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"kube-root-ca.crt\"" Oct 31 00:46:27.488531 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.487953 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"default-dockercfg-gs5fn\"" Oct 31 00:46:27.488531 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.487987 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"message-dumper-raw-233d7-predictor-serving-cert\"" Oct 31 00:46:27.571769 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.571731 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cvwb\" (UniqueName: \"kubernetes.io/projected/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-api-access-7cvwb\") pod \"message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh\" (UID: \"29925dd3-042a-4b35-8a03-7ae7e5b89359\") " pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:27.571769 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.571778 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh\" (UID: \"29925dd3-042a-4b35-8a03-7ae7e5b89359\") " pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:27.572004 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.571798 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29925dd3-042a-4b35-8a03-7ae7e5b89359-proxy-tls\") pod \"message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh\" (UID: \"29925dd3-042a-4b35-8a03-7ae7e5b89359\") " pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:27.672337 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.672311 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cvwb\" (UniqueName: \"kubernetes.io/projected/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-api-access-7cvwb\") pod \"message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh\" (UID: \"29925dd3-042a-4b35-8a03-7ae7e5b89359\") " pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:27.672453 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.672344 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh\" (UID: \"29925dd3-042a-4b35-8a03-7ae7e5b89359\") " pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:27.672453 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.672364 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29925dd3-042a-4b35-8a03-7ae7e5b89359-proxy-tls\") pod \"message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh\" (UID: \"29925dd3-042a-4b35-8a03-7ae7e5b89359\") " pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:27.673054 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.673028 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh\" (UID: \"29925dd3-042a-4b35-8a03-7ae7e5b89359\") " pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:27.674848 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.674826 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29925dd3-042a-4b35-8a03-7ae7e5b89359-proxy-tls\") pod \"message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh\" (UID: \"29925dd3-042a-4b35-8a03-7ae7e5b89359\") " pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:27.680951 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.680933 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cvwb\" (UniqueName: \"kubernetes.io/projected/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-api-access-7cvwb\") pod \"message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh\" (UID: \"29925dd3-042a-4b35-8a03-7ae7e5b89359\") " pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:27.797978 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:27.797877 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:28.067304 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:28.067231 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh"] Oct 31 00:46:28.070198 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:46:28.070174 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29925dd3_042a_4b35_8a03_7ae7e5b89359.slice/crio-4781e917b3923115882dfc5c46a5f6c2eb1c3bd06af5fe557d1a4883ae6d5563 WatchSource:0}: Error finding container 4781e917b3923115882dfc5c46a5f6c2eb1c3bd06af5fe557d1a4883ae6d5563: Status 404 returned error can't find the container with id 4781e917b3923115882dfc5c46a5f6c2eb1c3bd06af5fe557d1a4883ae6d5563 Oct 31 00:46:28.435315 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:28.435278 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" event={"ID":"29925dd3-042a-4b35-8a03-7ae7e5b89359","Type":"ContainerStarted","Data":"4781e917b3923115882dfc5c46a5f6c2eb1c3bd06af5fe557d1a4883ae6d5563"} Oct 31 00:46:29.440911 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:29.440880 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" event={"ID":"29925dd3-042a-4b35-8a03-7ae7e5b89359","Type":"ContainerStarted","Data":"93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9"} Oct 31 00:46:29.441298 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:29.440934 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" event={"ID":"29925dd3-042a-4b35-8a03-7ae7e5b89359","Type":"ContainerStarted","Data":"3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f"} Oct 31 00:46:29.441298 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:29.441041 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:29.459569 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:29.459525 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" podStartSLOduration=1.385001893 podStartE2EDuration="2.459511005s" totalImagesPullingTime="1.074509112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:29.458380823 +0000 UTC m=+972.819963465" watchObservedRunningTime="2026-10-31 00:46:29.459511005 +0000 UTC m=+972.821093705" Oct 31 00:46:30.445127 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:30.445095 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:30.446748 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:30.446727 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:34.139107 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.139073 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n"] Oct 31 00:46:34.142683 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.142658 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.148097 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.148074 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"isvc-secondary-f54d2d-kube-rbac-proxy-sar-config\"" Oct 31 00:46:34.150299 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.150282 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:46:34.150545 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.150496 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"openshift-service-ca.crt\"" Oct 31 00:46:34.152447 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.151607 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"isvc-secondary-f54d2d-predictor-serving-cert\"" Oct 31 00:46:34.152742 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.152722 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n"] Oct 31 00:46:34.152844 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.152756 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"invalid-s3-secret-f54d2d\"" Oct 31 00:46:34.152844 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.152838 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"invalid-s3-sa-f54d2d-dockercfg-v494w\"" Oct 31 00:46:34.226518 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.226475 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.226727 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.226535 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15a6be8-4bb4-4890-808c-e724156be986-kserve-provision-location\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.226727 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.226657 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b48sr\" (UniqueName: \"kubernetes.io/projected/c15a6be8-4bb4-4890-808c-e724156be986-kube-api-access-b48sr\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.226727 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.226700 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c15a6be8-4bb4-4890-808c-e724156be986-proxy-tls\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.226727 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.226722 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-cabundle-cert\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.248200 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.248179 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"kube-root-ca.crt\"" Oct 31 00:46:34.327361 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.327334 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15a6be8-4bb4-4890-808c-e724156be986-kserve-provision-location\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.327462 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.327384 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b48sr\" (UniqueName: \"kubernetes.io/projected/c15a6be8-4bb4-4890-808c-e724156be986-kube-api-access-b48sr\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.327462 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.327411 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c15a6be8-4bb4-4890-808c-e724156be986-proxy-tls\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.327462 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.327432 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-cabundle-cert\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.327462 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.327454 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.327807 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.327771 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15a6be8-4bb4-4890-808c-e724156be986-kserve-provision-location\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.328138 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.328104 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.328138 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.328131 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-cabundle-cert\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.330145 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.330125 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c15a6be8-4bb4-4890-808c-e724156be986-proxy-tls\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.336781 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.336757 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b48sr\" (UniqueName: \"kubernetes.io/projected/c15a6be8-4bb4-4890-808c-e724156be986-kube-api-access-b48sr\") pod \"isvc-secondary-f54d2d-predictor-8555cdb48-zg76n\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.466876 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.466843 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:34.738753 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:34.738726 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n"] Oct 31 00:46:34.741219 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:46:34.741190 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15a6be8_4bb4_4890_808c_e724156be986.slice/crio-b96aafd26755853f05b22f65627b9150c1cda2059d9093eee6f28291b46f71da WatchSource:0}: Error finding container b96aafd26755853f05b22f65627b9150c1cda2059d9093eee6f28291b46f71da: Status 404 returned error can't find the container with id b96aafd26755853f05b22f65627b9150c1cda2059d9093eee6f28291b46f71da Oct 31 00:46:35.463701 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:35.463666 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" event={"ID":"c15a6be8-4bb4-4890-808c-e724156be986","Type":"ContainerStarted","Data":"d913ed7141826b7d663c4b2d1d8e234a0552e67d1424593b71ca33873f98327f"} Oct 31 00:46:35.464185 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:35.463735 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" event={"ID":"c15a6be8-4bb4-4890-808c-e724156be986","Type":"ContainerStarted","Data":"b96aafd26755853f05b22f65627b9150c1cda2059d9093eee6f28291b46f71da"} Oct 31 00:46:37.455760 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:37.455731 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:46:41.486967 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:41.486933 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c950106202_isvc-secondary-f54d2d-predictor-8555cdb48-zg76n_c15a6be8-4bb4-4890-808c-e724156be986/storage-initializer/0.log" Oct 31 00:46:41.487321 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:41.486984 5821 generic.go:356] "Generic (PLEG): container finished" podID="c15a6be8-4bb4-4890-808c-e724156be986" containerID="d913ed7141826b7d663c4b2d1d8e234a0552e67d1424593b71ca33873f98327f" exitCode=1 Oct 31 00:46:41.487321 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:41.487048 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" event={"ID":"c15a6be8-4bb4-4890-808c-e724156be986","Type":"ContainerDied","Data":"d913ed7141826b7d663c4b2d1d8e234a0552e67d1424593b71ca33873f98327f"} Oct 31 00:46:42.492600 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:42.492576 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c950106202_isvc-secondary-f54d2d-predictor-8555cdb48-zg76n_c15a6be8-4bb4-4890-808c-e724156be986/storage-initializer/0.log" Oct 31 00:46:42.493006 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:42.492647 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" event={"ID":"c15a6be8-4bb4-4890-808c-e724156be986","Type":"ContainerStarted","Data":"02d0708aa86768e3019870c2869e25716536464bebf54e6ef746df33cbee61e3"} Oct 31 00:46:46.508401 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:46.508374 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c950106202_isvc-secondary-f54d2d-predictor-8555cdb48-zg76n_c15a6be8-4bb4-4890-808c-e724156be986/storage-initializer/1.log" Oct 31 00:46:46.508831 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:46.508745 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c950106202_isvc-secondary-f54d2d-predictor-8555cdb48-zg76n_c15a6be8-4bb4-4890-808c-e724156be986/storage-initializer/0.log" Oct 31 00:46:46.508831 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:46.508779 5821 generic.go:356] "Generic (PLEG): container finished" podID="c15a6be8-4bb4-4890-808c-e724156be986" containerID="02d0708aa86768e3019870c2869e25716536464bebf54e6ef746df33cbee61e3" exitCode=1 Oct 31 00:46:46.508912 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:46.508861 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" event={"ID":"c15a6be8-4bb4-4890-808c-e724156be986","Type":"ContainerDied","Data":"02d0708aa86768e3019870c2869e25716536464bebf54e6ef746df33cbee61e3"} Oct 31 00:46:46.508962 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:46.508948 5821 scope.go:122] "RemoveContainer" containerID="d913ed7141826b7d663c4b2d1d8e234a0552e67d1424593b71ca33873f98327f" Oct 31 00:46:46.509335 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:46.509308 5821 scope.go:122] "RemoveContainer" containerID="d913ed7141826b7d663c4b2d1d8e234a0552e67d1424593b71ca33873f98327f" Oct 31 00:46:46.519410 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:46:46.519388 5821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-f54d2d-predictor-8555cdb48-zg76n_e2e-gw1-c950106202(c15a6be8-4bb4-4890-808c-e724156be986)\"" pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" podUID="c15a6be8-4bb4-4890-808c-e724156be986" Oct 31 00:46:46.519599 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:46:46.519573 5821 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f54d2d-predictor-8555cdb48-zg76n_e2e-gw1-c950106202_c15a6be8-4bb4-4890-808c-e724156be986_0 in pod sandbox b96aafd26755853f05b22f65627b9150c1cda2059d9093eee6f28291b46f71da from index: no such id: 'd913ed7141826b7d663c4b2d1d8e234a0552e67d1424593b71ca33873f98327f'" containerID="d913ed7141826b7d663c4b2d1d8e234a0552e67d1424593b71ca33873f98327f" Oct 31 00:46:46.519670 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:46.519609 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d913ed7141826b7d663c4b2d1d8e234a0552e67d1424593b71ca33873f98327f"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f54d2d-predictor-8555cdb48-zg76n_e2e-gw1-c950106202_c15a6be8-4bb4-4890-808c-e724156be986_0 in pod sandbox b96aafd26755853f05b22f65627b9150c1cda2059d9093eee6f28291b46f71da from index: no such id: 'd913ed7141826b7d663c4b2d1d8e234a0552e67d1424593b71ca33873f98327f'" Oct 31 00:46:47.514076 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.514051 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c950106202_isvc-secondary-f54d2d-predictor-8555cdb48-zg76n_c15a6be8-4bb4-4890-808c-e724156be986/storage-initializer/1.log" Oct 31 00:46:47.566251 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.566215 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr"] Oct 31 00:46:47.570779 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.570759 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.574423 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.574402 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-logger-raw-233d7-kube-rbac-proxy-sar-config\"" Oct 31 00:46:47.574564 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.574438 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:46:47.574675 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.574658 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"seaweedfs-s3-creds\"" Oct 31 00:46:47.574784 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.574751 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-logger-raw-233d7-predictor-serving-cert\"" Oct 31 00:46:47.580185 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.580163 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr"] Oct 31 00:46:47.632530 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.632496 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3155bf71-a6e4-401b-a353-1ade47f89aba-kserve-provision-location\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.632530 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.632531 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkq75\" (UniqueName: \"kubernetes.io/projected/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-api-access-pkq75\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.632751 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.632550 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-cabundle-cert\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.632751 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.632692 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3155bf71-a6e4-401b-a353-1ade47f89aba-proxy-tls\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.632751 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.632738 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.733358 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.733325 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.733540 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.733377 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3155bf71-a6e4-401b-a353-1ade47f89aba-kserve-provision-location\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.733540 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.733395 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkq75\" (UniqueName: \"kubernetes.io/projected/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-api-access-pkq75\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.733540 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.733474 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-cabundle-cert\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.733743 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.733542 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3155bf71-a6e4-401b-a353-1ade47f89aba-proxy-tls\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.733837 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.733817 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3155bf71-a6e4-401b-a353-1ade47f89aba-kserve-provision-location\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.734131 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.734106 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.734131 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.734103 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-cabundle-cert\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.736006 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.735985 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3155bf71-a6e4-401b-a353-1ade47f89aba-proxy-tls\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.741712 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.741691 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkq75\" (UniqueName: \"kubernetes.io/projected/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-api-access-pkq75\") pod \"isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:47.891930 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:47.891845 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:48.166602 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:48.166576 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr"] Oct 31 00:46:48.169200 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:46:48.169165 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3155bf71_a6e4_401b_a353_1ade47f89aba.slice/crio-a932e1a4fcf5c05d02d9afdac54c9d7b95902cc8e512072200c478b436b13aa0 WatchSource:0}: Error finding container a932e1a4fcf5c05d02d9afdac54c9d7b95902cc8e512072200c478b436b13aa0: Status 404 returned error can't find the container with id a932e1a4fcf5c05d02d9afdac54c9d7b95902cc8e512072200c478b436b13aa0 Oct 31 00:46:48.519194 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:48.519114 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" event={"ID":"3155bf71-a6e4-401b-a353-1ade47f89aba","Type":"ContainerStarted","Data":"fbd8a29462bed17542fc835077dd11223e7f25170be035073b917d94102c1331"} Oct 31 00:46:48.519194 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:48.519175 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" event={"ID":"3155bf71-a6e4-401b-a353-1ade47f89aba","Type":"ContainerStarted","Data":"a932e1a4fcf5c05d02d9afdac54c9d7b95902cc8e512072200c478b436b13aa0"} Oct 31 00:46:49.837368 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:46:49.837334 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:46:52.249761 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.249733 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n"] Oct 31 00:46:52.377788 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.377769 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c950106202_isvc-secondary-f54d2d-predictor-8555cdb48-zg76n_c15a6be8-4bb4-4890-808c-e724156be986/storage-initializer/1.log" Oct 31 00:46:52.380077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.380059 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:52.464945 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.464915 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c15a6be8-4bb4-4890-808c-e724156be986-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c15a6be8-4bb4-4890-808c-e724156be986-proxy-tls\") pod \"c15a6be8-4bb4-4890-808c-e724156be986\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " Oct 31 00:46:52.465077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.464969 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15a6be8-4bb4-4890-808c-e724156be986-kube-api-access-b48sr\" (UniqueName: \"kubernetes.io/projected/c15a6be8-4bb4-4890-808c-e724156be986-kube-api-access-b48sr\") pod \"c15a6be8-4bb4-4890-808c-e724156be986\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " Oct 31 00:46:52.465077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.465010 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-cabundle-cert\") pod \"c15a6be8-4bb4-4890-808c-e724156be986\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " Oct 31 00:46:52.465077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.465030 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15a6be8-4bb4-4890-808c-e724156be986-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15a6be8-4bb4-4890-808c-e724156be986-kserve-provision-location\") pod \"c15a6be8-4bb4-4890-808c-e724156be986\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " Oct 31 00:46:52.465077 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.465070 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-kube-rbac-proxy-sar-config\") pod \"c15a6be8-4bb4-4890-808c-e724156be986\" (UID: \"c15a6be8-4bb4-4890-808c-e724156be986\") " Oct 31 00:46:52.465331 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.465273 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15a6be8-4bb4-4890-808c-e724156be986-kserve-provision-location" pod "c15a6be8-4bb4-4890-808c-e724156be986" (UID: "c15a6be8-4bb4-4890-808c-e724156be986"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:52.465405 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.465349 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-cabundle-cert" pod "c15a6be8-4bb4-4890-808c-e724156be986" (UID: "c15a6be8-4bb4-4890-808c-e724156be986"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:52.465462 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.465439 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-kube-rbac-proxy-sar-config" pod "c15a6be8-4bb4-4890-808c-e724156be986" (UID: "c15a6be8-4bb4-4890-808c-e724156be986"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:52.467118 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.467094 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15a6be8-4bb4-4890-808c-e724156be986-proxy-tls" pod "c15a6be8-4bb4-4890-808c-e724156be986" (UID: "c15a6be8-4bb4-4890-808c-e724156be986"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:52.467198 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.467128 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15a6be8-4bb4-4890-808c-e724156be986-kube-api-access-b48sr" pod "c15a6be8-4bb4-4890-808c-e724156be986" (UID: "c15a6be8-4bb4-4890-808c-e724156be986"). InnerVolumeSpecName "kube-api-access-b48sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:52.533868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.533850 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-c950106202_isvc-secondary-f54d2d-predictor-8555cdb48-zg76n_c15a6be8-4bb4-4890-808c-e724156be986/storage-initializer/1.log" Oct 31 00:46:52.533972 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.533952 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" Oct 31 00:46:52.533972 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.533952 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n" event={"ID":"c15a6be8-4bb4-4890-808c-e724156be986","Type":"ContainerDied","Data":"b96aafd26755853f05b22f65627b9150c1cda2059d9093eee6f28291b46f71da"} Oct 31 00:46:52.534074 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.534032 5821 scope.go:122] "RemoveContainer" containerID="02d0708aa86768e3019870c2869e25716536464bebf54e6ef746df33cbee61e3" Oct 31 00:46:52.536855 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.536827 5821 generic.go:356] "Generic (PLEG): container finished" podID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerID="fbd8a29462bed17542fc835077dd11223e7f25170be035073b917d94102c1331" exitCode=0 Oct 31 00:46:52.536957 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.536866 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" event={"ID":"3155bf71-a6e4-401b-a353-1ade47f89aba","Type":"ContainerDied","Data":"fbd8a29462bed17542fc835077dd11223e7f25170be035073b917d94102c1331"} Oct 31 00:46:52.565893 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.565867 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b48sr\" (UniqueName: \"kubernetes.io/projected/c15a6be8-4bb4-4890-808c-e724156be986-kube-api-access-b48sr\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:52.565998 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.565900 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:52.565998 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.565915 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15a6be8-4bb4-4890-808c-e724156be986-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:52.565998 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.565929 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c15a6be8-4bb4-4890-808c-e724156be986-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:52.565998 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.565942 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c15a6be8-4bb4-4890-808c-e724156be986-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:52.583232 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.583170 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n"] Oct 31 00:46:52.587648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.587592 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c950106202/isvc-secondary-f54d2d-predictor-8555cdb48-zg76n"] Oct 31 00:46:52.778321 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:52.778293 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15a6be8-4bb4-4890-808c-e724156be986" path="/var/lib/kubelet/pods/c15a6be8-4bb4-4890-808c-e724156be986/volumes" Oct 31 00:46:53.543815 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:53.543778 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" event={"ID":"3155bf71-a6e4-401b-a353-1ade47f89aba","Type":"ContainerStarted","Data":"ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367"} Oct 31 00:46:53.544264 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:53.543856 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" event={"ID":"3155bf71-a6e4-401b-a353-1ade47f89aba","Type":"ContainerStarted","Data":"9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3"} Oct 31 00:46:53.544264 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:53.543885 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" event={"ID":"3155bf71-a6e4-401b-a353-1ade47f89aba","Type":"ContainerStarted","Data":"4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940"} Oct 31 00:46:53.544264 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:53.544114 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:53.544264 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:53.544176 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:53.545712 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:53.545685 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:46:53.563821 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:53.563781 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podStartSLOduration=2.695655273 podStartE2EDuration="6.56376822s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.868112947s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:53.562940952 +0000 UTC m=+996.924523591" watchObservedRunningTime="2026-10-31 00:46:53.56376822 +0000 UTC m=+996.925350911" Oct 31 00:46:54.548275 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:54.548243 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:46:54.548777 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:54.548381 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:46:54.549469 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:54.549448 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:46:55.556688 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:55.556648 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:46:55.557075 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:46:55.557048 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:00.560762 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:00.560685 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:47:00.561217 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:00.561190 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:47:00.561567 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:00.561544 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:10.561229 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:10.561192 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:47:10.561757 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:10.561681 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:19.838108 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:47:19.838070 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:47:20.561913 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:20.561864 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:47:20.562333 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:20.562309 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:30.561283 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:30.561239 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:47:30.561825 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:30.561678 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:40.561864 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:40.561827 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:47:40.562395 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:40.562340 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:49.837600 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:47:49.837562 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:47:50.561740 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:50.561693 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:47:50.562128 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:47:50.562101 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:00.561954 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:00.561916 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:48:00.562549 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:00.562521 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:10.561784 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:10.561749 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:48:10.562395 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:10.562267 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:48:19.838047 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:48:19.837999 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:48:23.222466 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.222438 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-c89c0d682b_message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh_29925dd3-042a-4b35-8a03-7ae7e5b89359/kserve-container/0.log" Oct 31 00:48:23.524825 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.524749 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr"] Oct 31 00:48:23.525043 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.525006 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" containerID="cri-o://4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940" gracePeriod=30 Oct 31 00:48:23.525148 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.525046 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kube-rbac-proxy" containerID="cri-o://9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3" gracePeriod=30 Oct 31 00:48:23.525148 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.525045 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" containerID="cri-o://ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367" gracePeriod=30 Oct 31 00:48:23.553479 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.553451 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh"] Oct 31 00:48:23.553712 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.553691 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" podUID="29925dd3-042a-4b35-8a03-7ae7e5b89359" containerName="kserve-container" containerID="cri-o://3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f" gracePeriod=30 Oct 31 00:48:23.553847 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.553800 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" podUID="29925dd3-042a-4b35-8a03-7ae7e5b89359" containerName="kube-rbac-proxy" containerID="cri-o://93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9" gracePeriod=30 Oct 31 00:48:23.796557 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.796533 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:48:23.852911 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.852879 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-api-access-7cvwb\" (UniqueName: \"kubernetes.io/projected/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-api-access-7cvwb\") pod \"29925dd3-042a-4b35-8a03-7ae7e5b89359\" (UID: \"29925dd3-042a-4b35-8a03-7ae7e5b89359\") " Oct 31 00:48:23.853064 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.852928 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29925dd3-042a-4b35-8a03-7ae7e5b89359-proxy-tls\" (UniqueName: \"kubernetes.io/secret/29925dd3-042a-4b35-8a03-7ae7e5b89359-proxy-tls\") pod \"29925dd3-042a-4b35-8a03-7ae7e5b89359\" (UID: \"29925dd3-042a-4b35-8a03-7ae7e5b89359\") " Oct 31 00:48:23.853064 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.852993 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-rbac-proxy-sar-config\") pod \"29925dd3-042a-4b35-8a03-7ae7e5b89359\" (UID: \"29925dd3-042a-4b35-8a03-7ae7e5b89359\") " Oct 31 00:48:23.853361 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.853334 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-rbac-proxy-sar-config" pod "29925dd3-042a-4b35-8a03-7ae7e5b89359" (UID: "29925dd3-042a-4b35-8a03-7ae7e5b89359"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:23.855238 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.855216 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29925dd3-042a-4b35-8a03-7ae7e5b89359-proxy-tls" pod "29925dd3-042a-4b35-8a03-7ae7e5b89359" (UID: "29925dd3-042a-4b35-8a03-7ae7e5b89359"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:23.855321 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.855250 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-api-access-7cvwb" pod "29925dd3-042a-4b35-8a03-7ae7e5b89359" (UID: "29925dd3-042a-4b35-8a03-7ae7e5b89359"). InnerVolumeSpecName "kube-api-access-7cvwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:23.871868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.871839 5821 generic.go:356] "Generic (PLEG): container finished" podID="29925dd3-042a-4b35-8a03-7ae7e5b89359" containerID="93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9" exitCode=0 Oct 31 00:48:23.871868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.871865 5821 generic.go:356] "Generic (PLEG): container finished" podID="29925dd3-042a-4b35-8a03-7ae7e5b89359" containerID="3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f" exitCode=2 Oct 31 00:48:23.872059 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.871889 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" Oct 31 00:48:23.872059 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.871922 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" event={"ID":"29925dd3-042a-4b35-8a03-7ae7e5b89359","Type":"ContainerDied","Data":"93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9"} Oct 31 00:48:23.872059 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.871987 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" event={"ID":"29925dd3-042a-4b35-8a03-7ae7e5b89359","Type":"ContainerDied","Data":"3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f"} Oct 31 00:48:23.872059 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.872019 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh" event={"ID":"29925dd3-042a-4b35-8a03-7ae7e5b89359","Type":"ContainerDied","Data":"4781e917b3923115882dfc5c46a5f6c2eb1c3bd06af5fe557d1a4883ae6d5563"} Oct 31 00:48:23.872059 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.872059 5821 scope.go:122] "RemoveContainer" containerID="93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9" Oct 31 00:48:23.874544 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.874523 5821 generic.go:356] "Generic (PLEG): container finished" podID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerID="9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3" exitCode=0 Oct 31 00:48:23.874680 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.874552 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" event={"ID":"3155bf71-a6e4-401b-a353-1ade47f89aba","Type":"ContainerDied","Data":"9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3"} Oct 31 00:48:23.881270 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.881185 5821 scope.go:122] "RemoveContainer" containerID="3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f" Oct 31 00:48:23.890408 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.890391 5821 scope.go:122] "RemoveContainer" containerID="93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9" Oct 31 00:48:23.890690 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:48:23.890668 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9\": container with ID starting with 93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9 not found: ID does not exist" containerID="93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9" Oct 31 00:48:23.890801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.890696 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9"} err="failed to get container status \"93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9\": rpc error: code = NotFound desc = could not find container \"93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9\": container with ID starting with 93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9 not found: ID does not exist" Oct 31 00:48:23.890801 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.890711 5821 scope.go:122] "RemoveContainer" containerID="3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f" Oct 31 00:48:23.890994 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:48:23.890976 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f\": container with ID starting with 3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f not found: ID does not exist" containerID="3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f" Oct 31 00:48:23.891035 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.890999 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f"} err="failed to get container status \"3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f\": rpc error: code = NotFound desc = could not find container \"3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f\": container with ID starting with 3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f not found: ID does not exist" Oct 31 00:48:23.891035 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.891015 5821 scope.go:122] "RemoveContainer" containerID="93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9" Oct 31 00:48:23.891246 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.891229 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9"} err="failed to get container status \"93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9\": rpc error: code = NotFound desc = could not find container \"93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9\": container with ID starting with 93328b9f9a8329cb3aefc2f3d6ba4205b5ecb6dcf4239a65295d884713f1f8b9 not found: ID does not exist" Oct 31 00:48:23.891246 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.891246 5821 scope.go:122] "RemoveContainer" containerID="3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f" Oct 31 00:48:23.891502 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.891469 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f"} err="failed to get container status \"3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f\": rpc error: code = NotFound desc = could not find container \"3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f\": container with ID starting with 3dfdfcc0020c24fe0fa8b907ce70c14d5e197f7927ef8fd26d93890b49c96f3f not found: ID does not exist" Oct 31 00:48:23.894825 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.894802 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh"] Oct 31 00:48:23.899059 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.899040 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-c89c0d682b/message-dumper-raw-233d7-predictor-84b8bbd796-hlpnh"] Oct 31 00:48:23.953743 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.953709 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:23.953743 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.953739 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cvwb\" (UniqueName: \"kubernetes.io/projected/29925dd3-042a-4b35-8a03-7ae7e5b89359-kube-api-access-7cvwb\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:23.953922 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:23.953755 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29925dd3-042a-4b35-8a03-7ae7e5b89359-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:24.777727 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:24.777699 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29925dd3-042a-4b35-8a03-7ae7e5b89359" path="/var/lib/kubelet/pods/29925dd3-042a-4b35-8a03-7ae7e5b89359/volumes" Oct 31 00:48:25.557421 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:25.557379 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused" Oct 31 00:48:28.894649 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:28.894598 5821 generic.go:356] "Generic (PLEG): container finished" podID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerID="4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940" exitCode=0 Oct 31 00:48:28.894995 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:28.894669 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" event={"ID":"3155bf71-a6e4-401b-a353-1ade47f89aba","Type":"ContainerDied","Data":"4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940"} Oct 31 00:48:30.557550 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:30.557510 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused" Oct 31 00:48:30.561945 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:30.561924 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:48:30.563484 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:30.563457 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:35.557358 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:35.557314 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused" Oct 31 00:48:35.557838 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:35.557442 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:48:40.557479 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:40.557439 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused" Oct 31 00:48:40.561783 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:40.561763 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:48:40.563272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:40.563239 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:45.557567 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:45.557524 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused" Oct 31 00:48:49.837430 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:48:49.837392 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:48:50.556763 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:50.556724 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused" Oct 31 00:48:50.562066 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:50.562043 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:48:50.562206 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:50.562192 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:48:50.562734 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:50.562708 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:50.562844 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:50.562832 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:48:53.669945 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.669921 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:48:53.773415 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.773384 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3155bf71-a6e4-401b-a353-1ade47f89aba-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3155bf71-a6e4-401b-a353-1ade47f89aba-proxy-tls\") pod \"3155bf71-a6e4-401b-a353-1ade47f89aba\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " Oct 31 00:48:53.773616 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.773454 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-cabundle-cert\") pod \"3155bf71-a6e4-401b-a353-1ade47f89aba\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " Oct 31 00:48:53.773616 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.773476 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-api-access-pkq75\" (UniqueName: \"kubernetes.io/projected/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-api-access-pkq75\") pod \"3155bf71-a6e4-401b-a353-1ade47f89aba\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " Oct 31 00:48:53.773771 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.773657 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-rbac-proxy-sar-config\") pod \"3155bf71-a6e4-401b-a353-1ade47f89aba\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " Oct 31 00:48:53.773771 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.773706 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3155bf71-a6e4-401b-a353-1ade47f89aba-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3155bf71-a6e4-401b-a353-1ade47f89aba-kserve-provision-location\") pod \"3155bf71-a6e4-401b-a353-1ade47f89aba\" (UID: \"3155bf71-a6e4-401b-a353-1ade47f89aba\") " Oct 31 00:48:53.773866 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.773773 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-cabundle-cert" pod "3155bf71-a6e4-401b-a353-1ade47f89aba" (UID: "3155bf71-a6e4-401b-a353-1ade47f89aba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:53.774048 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.774017 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.774048 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.774019 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-rbac-proxy-sar-config" pod "3155bf71-a6e4-401b-a353-1ade47f89aba" (UID: "3155bf71-a6e4-401b-a353-1ade47f89aba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:53.774048 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.774030 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3155bf71-a6e4-401b-a353-1ade47f89aba-kserve-provision-location" pod "3155bf71-a6e4-401b-a353-1ade47f89aba" (UID: "3155bf71-a6e4-401b-a353-1ade47f89aba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.775788 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.775768 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3155bf71-a6e4-401b-a353-1ade47f89aba-proxy-tls" pod "3155bf71-a6e4-401b-a353-1ade47f89aba" (UID: "3155bf71-a6e4-401b-a353-1ade47f89aba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:53.775868 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.775797 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-api-access-pkq75" pod "3155bf71-a6e4-401b-a353-1ade47f89aba" (UID: "3155bf71-a6e4-401b-a353-1ade47f89aba"). InnerVolumeSpecName "kube-api-access-pkq75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:53.875199 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.875116 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkq75\" (UniqueName: \"kubernetes.io/projected/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-api-access-pkq75\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.875199 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.875150 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3155bf71-a6e4-401b-a353-1ade47f89aba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.875199 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.875164 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3155bf71-a6e4-401b-a353-1ade47f89aba-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.875199 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.875178 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3155bf71-a6e4-401b-a353-1ade47f89aba-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982187 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.982153 5821 generic.go:356] "Generic (PLEG): container finished" podID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerID="ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367" exitCode=0 Oct 31 00:48:53.982187 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.982190 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" event={"ID":"3155bf71-a6e4-401b-a353-1ade47f89aba","Type":"ContainerDied","Data":"ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367"} Oct 31 00:48:53.982378 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.982217 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" Oct 31 00:48:53.982378 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.982243 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr" event={"ID":"3155bf71-a6e4-401b-a353-1ade47f89aba","Type":"ContainerDied","Data":"a932e1a4fcf5c05d02d9afdac54c9d7b95902cc8e512072200c478b436b13aa0"} Oct 31 00:48:53.982378 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.982273 5821 scope.go:122] "RemoveContainer" containerID="ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367" Oct 31 00:48:53.991056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.991040 5821 scope.go:122] "RemoveContainer" containerID="9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3" Oct 31 00:48:53.998153 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:53.998135 5821 scope.go:122] "RemoveContainer" containerID="4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940" Oct 31 00:48:54.004664 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:54.004641 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr"] Oct 31 00:48:54.005817 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:54.005793 5821 scope.go:122] "RemoveContainer" containerID="fbd8a29462bed17542fc835077dd11223e7f25170be035073b917d94102c1331" Oct 31 00:48:54.009838 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:54.009820 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-logger-raw-233d7-predictor-9f7f5b694-22mdr"] Oct 31 00:48:54.013228 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:54.013208 5821 scope.go:122] "RemoveContainer" containerID="ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367" Oct 31 00:48:54.013513 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:48:54.013489 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367\": container with ID starting with ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367 not found: ID does not exist" containerID="ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367" Oct 31 00:48:54.013575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:54.013524 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367"} err="failed to get container status \"ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367\": rpc error: code = NotFound desc = could not find container \"ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367\": container with ID starting with ecffc5265633eb2687e05fe538ec6fe9c67d5d7b00a4ba0384c73a10c917f367 not found: ID does not exist" Oct 31 00:48:54.013575 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:54.013541 5821 scope.go:122] "RemoveContainer" containerID="9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3" Oct 31 00:48:54.013889 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:48:54.013872 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3\": container with ID starting with 9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3 not found: ID does not exist" containerID="9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3" Oct 31 00:48:54.013953 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:54.013895 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3"} err="failed to get container status \"9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3\": rpc error: code = NotFound desc = could not find container \"9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3\": container with ID starting with 9608396a2a30c166d40fc1ad7dc2a8ddd9fba255f31308b48b1810a4f4e34dd3 not found: ID does not exist" Oct 31 00:48:54.013953 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:54.013911 5821 scope.go:122] "RemoveContainer" containerID="4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940" Oct 31 00:48:54.014156 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:48:54.014140 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940\": container with ID starting with 4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940 not found: ID does not exist" containerID="4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940" Oct 31 00:48:54.014203 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:54.014160 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940"} err="failed to get container status \"4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940\": rpc error: code = NotFound desc = could not find container \"4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940\": container with ID starting with 4481ffdaad234f22ea374cb735ae77b602d292225363a7c3d752f3e82ed7c940 not found: ID does not exist" Oct 31 00:48:54.014203 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:54.014172 5821 scope.go:122] "RemoveContainer" containerID="fbd8a29462bed17542fc835077dd11223e7f25170be035073b917d94102c1331" Oct 31 00:48:54.014415 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:48:54.014400 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd8a29462bed17542fc835077dd11223e7f25170be035073b917d94102c1331\": container with ID starting with fbd8a29462bed17542fc835077dd11223e7f25170be035073b917d94102c1331 not found: ID does not exist" containerID="fbd8a29462bed17542fc835077dd11223e7f25170be035073b917d94102c1331" Oct 31 00:48:54.014454 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:54.014420 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd8a29462bed17542fc835077dd11223e7f25170be035073b917d94102c1331"} err="failed to get container status \"fbd8a29462bed17542fc835077dd11223e7f25170be035073b917d94102c1331\": rpc error: code = NotFound desc = could not find container \"fbd8a29462bed17542fc835077dd11223e7f25170be035073b917d94102c1331\": container with ID starting with fbd8a29462bed17542fc835077dd11223e7f25170be035073b917d94102c1331 not found: ID does not exist" Oct 31 00:48:54.777404 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:54.777374 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" path="/var/lib/kubelet/pods/3155bf71-a6e4-401b-a353-1ade47f89aba/volumes" Oct 31 00:48:55.328610 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328575 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7"] Oct 31 00:48:55.328861 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328848 5821 cpu_manager.go:414] "containerMap: removing container" podUID="29925dd3-042a-4b35-8a03-7ae7e5b89359" containerName="kube-rbac-proxy" Oct 31 00:48:55.328904 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328862 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29925dd3-042a-4b35-8a03-7ae7e5b89359" containerName="kube-rbac-proxy" Oct 31 00:48:55.328904 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328877 5821 cpu_manager.go:414] "containerMap: removing container" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="storage-initializer" Oct 31 00:48:55.328904 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328882 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="storage-initializer" Oct 31 00:48:55.328904 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328888 5821 cpu_manager.go:414] "containerMap: removing container" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" Oct 31 00:48:55.328904 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328895 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" Oct 31 00:48:55.328904 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328902 5821 cpu_manager.go:414] "containerMap: removing container" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kube-rbac-proxy" Oct 31 00:48:55.328904 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328906 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kube-rbac-proxy" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328920 5821 cpu_manager.go:414] "containerMap: removing container" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328925 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328932 5821 cpu_manager.go:414] "containerMap: removing container" podUID="c15a6be8-4bb4-4890-808c-e724156be986" containerName="storage-initializer" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328936 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15a6be8-4bb4-4890-808c-e724156be986" containerName="storage-initializer" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328942 5821 cpu_manager.go:414] "containerMap: removing container" podUID="c15a6be8-4bb4-4890-808c-e724156be986" containerName="storage-initializer" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328946 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15a6be8-4bb4-4890-808c-e724156be986" containerName="storage-initializer" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328962 5821 cpu_manager.go:414] "containerMap: removing container" podUID="29925dd3-042a-4b35-8a03-7ae7e5b89359" containerName="kserve-container" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.328967 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29925dd3-042a-4b35-8a03-7ae7e5b89359" containerName="kserve-container" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.329004 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="29925dd3-042a-4b35-8a03-7ae7e5b89359" containerName="kserve-container" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.329012 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="29925dd3-042a-4b35-8a03-7ae7e5b89359" containerName="kube-rbac-proxy" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.329018 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="agent" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.329022 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kube-rbac-proxy" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.329029 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="3155bf71-a6e4-401b-a353-1ade47f89aba" containerName="kserve-container" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.329035 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15a6be8-4bb4-4890-808c-e724156be986" containerName="storage-initializer" Oct 31 00:48:55.329111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.329039 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15a6be8-4bb4-4890-808c-e724156be986" containerName="storage-initializer" Oct 31 00:48:55.333683 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.333666 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.338048 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.338027 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"kube-root-ca.crt\"" Oct 31 00:48:55.340432 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.340409 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:48:55.340566 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.340465 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"openshift-service-ca.crt\"" Oct 31 00:48:55.341301 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.341280 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-sklearn-scale-raw-27e78-predictor-serving-cert\"" Oct 31 00:48:55.341416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.341398 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"seaweedfs-s3-creds\"" Oct 31 00:48:55.341511 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.341417 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-c89c0d682b\"/\"default-dockercfg-gs5fn\"" Oct 31 00:48:55.342369 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.342341 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7"] Oct 31 00:48:55.385264 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.385232 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab2ddf52-6843-4245-9efa-cb565bf75a74-proxy-tls\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.385264 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.385265 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab2ddf52-6843-4245-9efa-cb565bf75a74-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.385495 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.385285 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-cabundle-cert\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.385495 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.385360 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.385611 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.385536 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpjjt\" (UniqueName: \"kubernetes.io/projected/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-api-access-rpjjt\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.438966 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.438937 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-c89c0d682b\"/\"isvc-sklearn-scale-raw-27e78-kube-rbac-proxy-sar-config\"" Oct 31 00:48:55.486496 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.486466 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpjjt\" (UniqueName: \"kubernetes.io/projected/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-api-access-rpjjt\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.486613 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.486509 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab2ddf52-6843-4245-9efa-cb565bf75a74-proxy-tls\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.486613 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.486526 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab2ddf52-6843-4245-9efa-cb565bf75a74-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.486613 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.486542 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-cabundle-cert\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.486613 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.486587 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.486986 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.486963 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab2ddf52-6843-4245-9efa-cb565bf75a74-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.487291 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.487266 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-cabundle-cert\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.487360 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.487277 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.489155 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.489133 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab2ddf52-6843-4245-9efa-cb565bf75a74-proxy-tls\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.495059 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.495039 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpjjt\" (UniqueName: \"kubernetes.io/projected/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-api-access-rpjjt\") pod \"isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.651506 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.651423 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:48:55.921873 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.921844 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7"] Oct 31 00:48:55.925118 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:48:55.925090 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2ddf52_6843_4245_9efa_cb565bf75a74.slice/crio-5c354f9adfe9f9cd460901478ebd7b26e00b9a8950179c65423f75640d494b16 WatchSource:0}: Error finding container 5c354f9adfe9f9cd460901478ebd7b26e00b9a8950179c65423f75640d494b16: Status 404 returned error can't find the container with id 5c354f9adfe9f9cd460901478ebd7b26e00b9a8950179c65423f75640d494b16 Oct 31 00:48:55.991425 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.991396 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" event={"ID":"ab2ddf52-6843-4245-9efa-cb565bf75a74","Type":"ContainerStarted","Data":"1cb2d9c44e4db1850a6ca99c4d5b283c72e173bca3985882ecc577bcb50a49f2"} Oct 31 00:48:55.991528 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:48:55.991461 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" event={"ID":"ab2ddf52-6843-4245-9efa-cb565bf75a74","Type":"ContainerStarted","Data":"5c354f9adfe9f9cd460901478ebd7b26e00b9a8950179c65423f75640d494b16"} Oct 31 00:49:00.009477 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:00.009444 5821 generic.go:356] "Generic (PLEG): container finished" podID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerID="1cb2d9c44e4db1850a6ca99c4d5b283c72e173bca3985882ecc577bcb50a49f2" exitCode=0 Oct 31 00:49:00.009967 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:00.009517 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" event={"ID":"ab2ddf52-6843-4245-9efa-cb565bf75a74","Type":"ContainerDied","Data":"1cb2d9c44e4db1850a6ca99c4d5b283c72e173bca3985882ecc577bcb50a49f2"} Oct 31 00:49:01.015662 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:01.015608 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" event={"ID":"ab2ddf52-6843-4245-9efa-cb565bf75a74","Type":"ContainerStarted","Data":"f7e740607321d435d54a0dcae4ab49cb82525f392a7d89396894398930b743cc"} Oct 31 00:49:01.016096 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:01.015692 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" event={"ID":"ab2ddf52-6843-4245-9efa-cb565bf75a74","Type":"ContainerStarted","Data":"51a09663301e1530d8dd18f4d2a609543b259f27d3a8b955dda81c9fddafce07"} Oct 31 00:49:01.016096 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:01.016033 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:49:01.016096 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:01.016083 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:49:01.017403 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:01.017377 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:49:01.035432 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:01.035390 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podStartSLOduration=2.079340625 podStartE2EDuration="6.035379472s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.956038847s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:01.034396027 +0000 UTC m=+1124.395978699" watchObservedRunningTime="2026-10-31 00:49:01.035379472 +0000 UTC m=+1124.396962111" Oct 31 00:49:02.019282 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:02.019244 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:49:07.023675 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:07.023645 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:49:07.024121 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:07.024098 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:49:17.024465 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:17.024431 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:49:19.838137 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:49:19.838097 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:49:27.024050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:27.024012 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:49:37.024108 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:37.024071 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:49:47.024140 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:47.024097 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:49:49.838296 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:49:49.838257 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:49:57.024126 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:49:57.024083 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:50:07.024875 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:07.024793 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:50:11.773725 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:11.773686 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:50:19.837828 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:50:19.837787 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:50:21.777563 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:21.777512 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:50:26.082154 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.082118 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc"] Oct 31 00:50:26.085164 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.085142 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.096958 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.096933 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc"] Oct 31 00:50:26.178364 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.178338 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-cabundle-cert\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.178523 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.178371 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd0c741-5942-4a47-8936-d8c323069ad4-kserve-provision-location\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.178523 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.178461 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzlmz\" (UniqueName: \"kubernetes.io/projected/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-api-access-wzlmz\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.178523 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.178500 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.178653 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.178535 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd0c741-5942-4a47-8936-d8c323069ad4-proxy-tls\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.184754 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.184726 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:50:26.184886 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.184774 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"seaweedfs-s3-creds\"" Oct 31 00:50:26.184886 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.184775 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"openshift-service-ca.crt\"" Oct 31 00:50:26.184886 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.184792 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"isvc-canary-create-predictor-serving-cert\"" Oct 31 00:50:26.184886 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.184792 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"kube-root-ca.crt\"" Oct 31 00:50:26.184886 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.184835 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"default-dockercfg-84b4v\"" Oct 31 00:50:26.184886 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.184838 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\"" Oct 31 00:50:26.279190 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.279159 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.279317 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.279206 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd0c741-5942-4a47-8936-d8c323069ad4-proxy-tls\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.279317 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.279229 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-cabundle-cert\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.279426 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.279340 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd0c741-5942-4a47-8936-d8c323069ad4-kserve-provision-location\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.279469 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.279425 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzlmz\" (UniqueName: \"kubernetes.io/projected/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-api-access-wzlmz\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.279805 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.279785 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd0c741-5942-4a47-8936-d8c323069ad4-kserve-provision-location\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.279870 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.279816 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.279947 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.279931 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-cabundle-cert\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.281840 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.281822 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd0c741-5942-4a47-8936-d8c323069ad4-proxy-tls\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.287692 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.287671 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzlmz\" (UniqueName: \"kubernetes.io/projected/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-api-access-wzlmz\") pod \"isvc-canary-create-predictor-6b447db959-6mbnc\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.402785 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.402703 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:26.671187 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:26.671159 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc"] Oct 31 00:50:26.674253 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:50:26.674215 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd0c741_5942_4a47_8936_d8c323069ad4.slice/crio-a1a152554501df1a303d6afbbc0444d8f6f3a2f1791cb686839bd4454e45a050 WatchSource:0}: Error finding container a1a152554501df1a303d6afbbc0444d8f6f3a2f1791cb686839bd4454e45a050: Status 404 returned error can't find the container with id a1a152554501df1a303d6afbbc0444d8f6f3a2f1791cb686839bd4454e45a050 Oct 31 00:50:27.321435 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:27.321398 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" event={"ID":"3cd0c741-5942-4a47-8936-d8c323069ad4","Type":"ContainerStarted","Data":"2c6d18c8da040fe2f2ff96f6860977ffbad26388aa8eb55e5fe1493f9c9078b7"} Oct 31 00:50:27.321932 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:27.321492 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" event={"ID":"3cd0c741-5942-4a47-8936-d8c323069ad4","Type":"ContainerStarted","Data":"a1a152554501df1a303d6afbbc0444d8f6f3a2f1791cb686839bd4454e45a050"} Oct 31 00:50:31.336249 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:31.336219 5821 generic.go:356] "Generic (PLEG): container finished" podID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerID="2c6d18c8da040fe2f2ff96f6860977ffbad26388aa8eb55e5fe1493f9c9078b7" exitCode=0 Oct 31 00:50:31.336638 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:31.336289 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" event={"ID":"3cd0c741-5942-4a47-8936-d8c323069ad4","Type":"ContainerDied","Data":"2c6d18c8da040fe2f2ff96f6860977ffbad26388aa8eb55e5fe1493f9c9078b7"} Oct 31 00:50:31.774234 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:31.774197 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:50:32.341589 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:32.341559 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" event={"ID":"3cd0c741-5942-4a47-8936-d8c323069ad4","Type":"ContainerStarted","Data":"067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90"} Oct 31 00:50:32.342041 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:32.341616 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" event={"ID":"3cd0c741-5942-4a47-8936-d8c323069ad4","Type":"ContainerStarted","Data":"3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca"} Oct 31 00:50:32.342041 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:32.341821 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:32.361460 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:32.361414 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podStartSLOduration=2.6147957550000003 podStartE2EDuration="6.361402908s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.746607153s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:32.359546052 +0000 UTC m=+1215.721128702" watchObservedRunningTime="2026-10-31 00:50:32.361402908 +0000 UTC m=+1215.722985546" Oct 31 00:50:33.345191 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:33.345154 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:33.346474 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:33.346448 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 31 00:50:34.348535 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:34.348497 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 31 00:50:39.352997 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:39.352967 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:50:39.353504 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:39.353448 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 31 00:50:41.774520 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:41.774480 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:50:49.354101 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:49.354064 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 31 00:50:49.836902 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:50:49.836865 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:50:51.773930 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:51.773888 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:50:59.353509 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:50:59.353466 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 31 00:51:01.774104 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:01.774067 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:51:09.354266 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:09.354226 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 31 00:51:11.774208 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:11.774167 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:51:19.354168 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:19.354131 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 31 00:51:19.837466 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:51:19.837431 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:51:21.773910 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:21.773874 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:51:29.353683 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:29.353580 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 31 00:51:31.774037 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:31.774004 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:51:38.777455 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:38.777426 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:51:39.354320 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:39.354278 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 31 00:51:47.240763 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:47.240733 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7"] Oct 31 00:51:47.241275 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:47.241046 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" containerID="cri-o://51a09663301e1530d8dd18f4d2a609543b259f27d3a8b955dda81c9fddafce07" gracePeriod=30 Oct 31 00:51:47.241275 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:47.241058 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kube-rbac-proxy" containerID="cri-o://f7e740607321d435d54a0dcae4ab49cb82525f392a7d89396894398930b743cc" gracePeriod=30 Oct 31 00:51:47.607928 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:47.607847 5821 generic.go:356] "Generic (PLEG): container finished" podID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerID="f7e740607321d435d54a0dcae4ab49cb82525f392a7d89396894398930b743cc" exitCode=0 Oct 31 00:51:47.607928 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:47.607889 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" event={"ID":"ab2ddf52-6843-4245-9efa-cb565bf75a74","Type":"ContainerDied","Data":"f7e740607321d435d54a0dcae4ab49cb82525f392a7d89396894398930b743cc"} Oct 31 00:51:48.773738 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:48.773703 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 31 00:51:49.014240 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:49.014205 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc"] Oct 31 00:51:49.014462 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:49.014441 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" containerID="cri-o://3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca" gracePeriod=30 Oct 31 00:51:49.014548 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:49.014504 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kube-rbac-proxy" containerID="cri-o://067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90" gracePeriod=30 Oct 31 00:51:49.015449 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:49.015429 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:51:49.349244 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:49.349146 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused" Oct 31 00:51:49.616357 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:49.616277 5821 generic.go:356] "Generic (PLEG): container finished" podID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerID="067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90" exitCode=0 Oct 31 00:51:49.616357 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:49.616336 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" event={"ID":"3cd0c741-5942-4a47-8936-d8c323069ad4","Type":"ContainerDied","Data":"067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90"} Oct 31 00:51:49.837895 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:51:49.837862 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:51:52.020015 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:52.019972 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 31 00:51:53.955878 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:53.955854 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:51:54.063370 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.063284 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-rbac-proxy-sar-config\") pod \"3cd0c741-5942-4a47-8936-d8c323069ad4\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " Oct 31 00:51:54.063370 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.063335 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd0c741-5942-4a47-8936-d8c323069ad4-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd0c741-5942-4a47-8936-d8c323069ad4-kserve-provision-location\") pod \"3cd0c741-5942-4a47-8936-d8c323069ad4\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " Oct 31 00:51:54.063656 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.063395 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cd0c741-5942-4a47-8936-d8c323069ad4-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd0c741-5942-4a47-8936-d8c323069ad4-proxy-tls\") pod \"3cd0c741-5942-4a47-8936-d8c323069ad4\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " Oct 31 00:51:54.063656 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.063418 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-cabundle-cert\") pod \"3cd0c741-5942-4a47-8936-d8c323069ad4\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " Oct 31 00:51:54.063656 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.063449 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-api-access-wzlmz\" (UniqueName: \"kubernetes.io/projected/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-api-access-wzlmz\") pod \"3cd0c741-5942-4a47-8936-d8c323069ad4\" (UID: \"3cd0c741-5942-4a47-8936-d8c323069ad4\") " Oct 31 00:51:54.063820 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.063735 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-cabundle-cert" pod "3cd0c741-5942-4a47-8936-d8c323069ad4" (UID: "3cd0c741-5942-4a47-8936-d8c323069ad4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:54.063820 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.063749 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd0c741-5942-4a47-8936-d8c323069ad4-kserve-provision-location" pod "3cd0c741-5942-4a47-8936-d8c323069ad4" (UID: "3cd0c741-5942-4a47-8936-d8c323069ad4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:54.063820 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.063755 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-rbac-proxy-sar-config" pod "3cd0c741-5942-4a47-8936-d8c323069ad4" (UID: "3cd0c741-5942-4a47-8936-d8c323069ad4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:54.065712 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.065689 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-api-access-wzlmz" pod "3cd0c741-5942-4a47-8936-d8c323069ad4" (UID: "3cd0c741-5942-4a47-8936-d8c323069ad4"). InnerVolumeSpecName "kube-api-access-wzlmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:54.065899 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.065880 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd0c741-5942-4a47-8936-d8c323069ad4-proxy-tls" pod "3cd0c741-5942-4a47-8936-d8c323069ad4" (UID: "3cd0c741-5942-4a47-8936-d8c323069ad4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:54.164408 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.164379 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd0c741-5942-4a47-8936-d8c323069ad4-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:54.164408 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.164404 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:54.164408 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.164414 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzlmz\" (UniqueName: \"kubernetes.io/projected/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-api-access-wzlmz\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:54.164612 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.164424 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd0c741-5942-4a47-8936-d8c323069ad4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:54.164612 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.164454 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd0c741-5942-4a47-8936-d8c323069ad4-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:54.636180 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.636143 5821 generic.go:356] "Generic (PLEG): container finished" podID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerID="3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca" exitCode=0 Oct 31 00:51:54.636358 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.636215 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" Oct 31 00:51:54.636358 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.636229 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" event={"ID":"3cd0c741-5942-4a47-8936-d8c323069ad4","Type":"ContainerDied","Data":"3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca"} Oct 31 00:51:54.636358 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.636322 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc" event={"ID":"3cd0c741-5942-4a47-8936-d8c323069ad4","Type":"ContainerDied","Data":"a1a152554501df1a303d6afbbc0444d8f6f3a2f1791cb686839bd4454e45a050"} Oct 31 00:51:54.636358 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.636350 5821 scope.go:122] "RemoveContainer" containerID="067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90" Oct 31 00:51:54.645110 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.645096 5821 scope.go:122] "RemoveContainer" containerID="3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca" Oct 31 00:51:54.652226 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.652209 5821 scope.go:122] "RemoveContainer" containerID="2c6d18c8da040fe2f2ff96f6860977ffbad26388aa8eb55e5fe1493f9c9078b7" Oct 31 00:51:54.656951 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.656929 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc"] Oct 31 00:51:54.660507 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.660488 5821 scope.go:122] "RemoveContainer" containerID="067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90" Oct 31 00:51:54.660948 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:51:54.660814 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90\": container with ID starting with 067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90 not found: ID does not exist" containerID="067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90" Oct 31 00:51:54.660948 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.660850 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-create-predictor-6b447db959-6mbnc"] Oct 31 00:51:54.660948 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.660856 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90"} err="failed to get container status \"067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90\": rpc error: code = NotFound desc = could not find container \"067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90\": container with ID starting with 067cbe7654f245b1b510bd806ec69d63c1fa3d825b31ad46031a5ef9e5bf5c90 not found: ID does not exist" Oct 31 00:51:54.660948 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.660881 5821 scope.go:122] "RemoveContainer" containerID="3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca" Oct 31 00:51:54.661164 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:51:54.661146 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca\": container with ID starting with 3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca not found: ID does not exist" containerID="3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca" Oct 31 00:51:54.661214 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.661172 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca"} err="failed to get container status \"3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca\": rpc error: code = NotFound desc = could not find container \"3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca\": container with ID starting with 3ccab8f8945baf3aa1db0813794c6817ac32e186c1ff285f3a818fd649b36eca not found: ID does not exist" Oct 31 00:51:54.661214 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.661187 5821 scope.go:122] "RemoveContainer" containerID="2c6d18c8da040fe2f2ff96f6860977ffbad26388aa8eb55e5fe1493f9c9078b7" Oct 31 00:51:54.661424 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:51:54.661406 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6d18c8da040fe2f2ff96f6860977ffbad26388aa8eb55e5fe1493f9c9078b7\": container with ID starting with 2c6d18c8da040fe2f2ff96f6860977ffbad26388aa8eb55e5fe1493f9c9078b7 not found: ID does not exist" containerID="2c6d18c8da040fe2f2ff96f6860977ffbad26388aa8eb55e5fe1493f9c9078b7" Oct 31 00:51:54.661468 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.661429 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6d18c8da040fe2f2ff96f6860977ffbad26388aa8eb55e5fe1493f9c9078b7"} err="failed to get container status \"2c6d18c8da040fe2f2ff96f6860977ffbad26388aa8eb55e5fe1493f9c9078b7\": rpc error: code = NotFound desc = could not find container \"2c6d18c8da040fe2f2ff96f6860977ffbad26388aa8eb55e5fe1493f9c9078b7\": container with ID starting with 2c6d18c8da040fe2f2ff96f6860977ffbad26388aa8eb55e5fe1493f9c9078b7 not found: ID does not exist" Oct 31 00:51:54.778273 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:54.778245 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" path="/var/lib/kubelet/pods/3cd0c741-5942-4a47-8936-d8c323069ad4/volumes" Oct 31 00:51:57.019890 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.019852 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 31 00:51:57.649201 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.649171 5821 generic.go:356] "Generic (PLEG): container finished" podID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerID="51a09663301e1530d8dd18f4d2a609543b259f27d3a8b955dda81c9fddafce07" exitCode=0 Oct 31 00:51:57.649357 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.649243 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" event={"ID":"ab2ddf52-6843-4245-9efa-cb565bf75a74","Type":"ContainerDied","Data":"51a09663301e1530d8dd18f4d2a609543b259f27d3a8b955dda81c9fddafce07"} Oct 31 00:51:57.682330 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.682307 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:51:57.790104 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.790021 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-api-access-rpjjt\" (UniqueName: \"kubernetes.io/projected/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-api-access-rpjjt\") pod \"ab2ddf52-6843-4245-9efa-cb565bf75a74\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " Oct 31 00:51:57.790104 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.790072 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab2ddf52-6843-4245-9efa-cb565bf75a74-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab2ddf52-6843-4245-9efa-cb565bf75a74-proxy-tls\") pod \"ab2ddf52-6843-4245-9efa-cb565bf75a74\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " Oct 31 00:51:57.790104 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.790105 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-cabundle-cert\") pod \"ab2ddf52-6843-4245-9efa-cb565bf75a74\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " Oct 31 00:51:57.790401 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.790141 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2ddf52-6843-4245-9efa-cb565bf75a74-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab2ddf52-6843-4245-9efa-cb565bf75a74-kserve-provision-location\") pod \"ab2ddf52-6843-4245-9efa-cb565bf75a74\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " Oct 31 00:51:57.790401 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.790164 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-rbac-proxy-sar-config\") pod \"ab2ddf52-6843-4245-9efa-cb565bf75a74\" (UID: \"ab2ddf52-6843-4245-9efa-cb565bf75a74\") " Oct 31 00:51:57.790512 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.790494 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2ddf52-6843-4245-9efa-cb565bf75a74-kserve-provision-location" pod "ab2ddf52-6843-4245-9efa-cb565bf75a74" (UID: "ab2ddf52-6843-4245-9efa-cb565bf75a74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:57.790571 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.790530 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-rbac-proxy-sar-config" pod "ab2ddf52-6843-4245-9efa-cb565bf75a74" (UID: "ab2ddf52-6843-4245-9efa-cb565bf75a74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:57.790571 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.790540 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-cabundle-cert" pod "ab2ddf52-6843-4245-9efa-cb565bf75a74" (UID: "ab2ddf52-6843-4245-9efa-cb565bf75a74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:57.792417 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.792383 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2ddf52-6843-4245-9efa-cb565bf75a74-proxy-tls" pod "ab2ddf52-6843-4245-9efa-cb565bf75a74" (UID: "ab2ddf52-6843-4245-9efa-cb565bf75a74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:57.792417 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.792415 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-api-access-rpjjt" pod "ab2ddf52-6843-4245-9efa-cb565bf75a74" (UID: "ab2ddf52-6843-4245-9efa-cb565bf75a74"). InnerVolumeSpecName "kube-api-access-rpjjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:57.891644 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.891591 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:57.891644 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.891628 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab2ddf52-6843-4245-9efa-cb565bf75a74-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:57.891644 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.891641 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:57.891644 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.891650 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpjjt\" (UniqueName: \"kubernetes.io/projected/ab2ddf52-6843-4245-9efa-cb565bf75a74-kube-api-access-rpjjt\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:57.891644 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:57.891660 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab2ddf52-6843-4245-9efa-cb565bf75a74-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:58.659347 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:58.659313 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" event={"ID":"ab2ddf52-6843-4245-9efa-cb565bf75a74","Type":"ContainerDied","Data":"5c354f9adfe9f9cd460901478ebd7b26e00b9a8950179c65423f75640d494b16"} Oct 31 00:51:58.659768 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:58.659357 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7" Oct 31 00:51:58.659768 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:58.659412 5821 scope.go:122] "RemoveContainer" containerID="f7e740607321d435d54a0dcae4ab49cb82525f392a7d89396894398930b743cc" Oct 31 00:51:58.667426 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:58.667409 5821 scope.go:122] "RemoveContainer" containerID="51a09663301e1530d8dd18f4d2a609543b259f27d3a8b955dda81c9fddafce07" Oct 31 00:51:58.674791 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:58.674771 5821 scope.go:122] "RemoveContainer" containerID="1cb2d9c44e4db1850a6ca99c4d5b283c72e173bca3985882ecc577bcb50a49f2" Oct 31 00:51:58.680845 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:58.680827 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7"] Oct 31 00:51:58.684683 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:58.684664 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-c89c0d682b/isvc-sklearn-scale-raw-27e78-predictor-69bcbfbb48-2mpb7"] Oct 31 00:51:58.778230 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:51:58.778200 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" path="/var/lib/kubelet/pods/ab2ddf52-6843-4245-9efa-cb565bf75a74/volumes" Oct 31 00:52:02.888291 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888248 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv"] Oct 31 00:52:02.888695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888536 5821 cpu_manager.go:414] "containerMap: removing container" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="storage-initializer" Oct 31 00:52:02.888695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888548 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="storage-initializer" Oct 31 00:52:02.888695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888559 5821 cpu_manager.go:414] "containerMap: removing container" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="storage-initializer" Oct 31 00:52:02.888695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888564 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="storage-initializer" Oct 31 00:52:02.888695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888578 5821 cpu_manager.go:414] "containerMap: removing container" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kube-rbac-proxy" Oct 31 00:52:02.888695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888586 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kube-rbac-proxy" Oct 31 00:52:02.888695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888599 5821 cpu_manager.go:414] "containerMap: removing container" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" Oct 31 00:52:02.888695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888603 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" Oct 31 00:52:02.888695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888611 5821 cpu_manager.go:414] "containerMap: removing container" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kube-rbac-proxy" Oct 31 00:52:02.888695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888616 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kube-rbac-proxy" Oct 31 00:52:02.888695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888641 5821 cpu_manager.go:414] "containerMap: removing container" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" Oct 31 00:52:02.888695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888645 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" Oct 31 00:52:02.889055 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888707 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kserve-container" Oct 31 00:52:02.889055 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888714 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd0c741-5942-4a47-8936-d8c323069ad4" containerName="kube-rbac-proxy" Oct 31 00:52:02.889055 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888720 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kserve-container" Oct 31 00:52:02.889055 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.888731 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2ddf52-6843-4245-9efa-cb565bf75a74" containerName="kube-rbac-proxy" Oct 31 00:52:02.893745 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.893724 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:02.898533 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.898511 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"seaweedfs-s3-creds\"" Oct 31 00:52:02.898690 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.898545 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Oct 31 00:52:02.898690 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.898597 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"openshift-service-ca.crt\"" Oct 31 00:52:02.898690 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.898609 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"kube-root-ca.crt\"" Oct 31 00:52:02.898690 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.898684 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Oct 31 00:52:02.898904 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.898755 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"default-dockercfg-84b4v\"" Oct 31 00:52:02.898904 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.898813 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-c950106202\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:52:02.902401 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.902382 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv"] Oct 31 00:52:02.949232 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.949201 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq"] Oct 31 00:52:02.952394 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.952378 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:02.962054 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.962032 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq"] Oct 31 00:52:02.998313 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:02.998284 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-c950106202\"/\"isvc-canary-promote-predictor-serving-cert\"" Oct 31 00:52:03.027204 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.027182 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.027323 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.027212 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86478869-cabb-4bd6-9204-3e6b34d189ab-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.027323 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.027235 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86478869-cabb-4bd6-9204-3e6b34d189ab-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.027394 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.027379 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.027428 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.027408 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zfzd\" (UniqueName: \"kubernetes.io/projected/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-api-access-8zfzd\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.128192 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.128154 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c0ca28-c9be-4eea-8305-a87d12262907-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.128342 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.128207 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.128342 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.128271 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86478869-cabb-4bd6-9204-3e6b34d189ab-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.128342 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.128294 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86478869-cabb-4bd6-9204-3e6b34d189ab-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.128342 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.128336 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.128549 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.128389 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.128549 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.128415 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6c0ca28-c9be-4eea-8305-a87d12262907-proxy-tls\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.128549 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.128437 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjkj\" (UniqueName: \"kubernetes.io/projected/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-api-access-nzjkj\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.128701 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.128592 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfzd\" (UniqueName: \"kubernetes.io/projected/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-api-access-8zfzd\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.128701 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.128654 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-cabundle-cert\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.128781 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.128751 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86478869-cabb-4bd6-9204-3e6b34d189ab-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.129141 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.129119 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.129338 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.129323 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.130992 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.130972 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86478869-cabb-4bd6-9204-3e6b34d189ab-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.136571 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.136551 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zfzd\" (UniqueName: \"kubernetes.io/projected/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-api-access-8zfzd\") pod \"isvc-canary-promote-v2-predictor-5997987886-lqjsv\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.211748 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.211718 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:03.229386 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.229358 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c0ca28-c9be-4eea-8305-a87d12262907-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.229514 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.229423 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.229514 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.229472 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6c0ca28-c9be-4eea-8305-a87d12262907-proxy-tls\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.229514 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.229496 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjkj\" (UniqueName: \"kubernetes.io/projected/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-api-access-nzjkj\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.229717 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.229527 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-cabundle-cert\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.229774 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.229748 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c0ca28-c9be-4eea-8305-a87d12262907-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.230118 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.230095 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.230224 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.230194 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-cabundle-cert\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.232637 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.232600 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6c0ca28-c9be-4eea-8305-a87d12262907-proxy-tls\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.237829 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.237805 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjkj\" (UniqueName: \"kubernetes.io/projected/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-api-access-nzjkj\") pod \"isvc-canary-promote-predictor-5cdb4c96cc-2hrwq\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.269012 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.268985 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:03.487082 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.487001 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv"] Oct 31 00:52:03.490438 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:52:03.490410 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86478869_cabb_4bd6_9204_3e6b34d189ab.slice/crio-c883fb82aaaa1e8d803aa0869c361c7b740411aeb33eb01faa3c46bc4772960d WatchSource:0}: Error finding container c883fb82aaaa1e8d803aa0869c361c7b740411aeb33eb01faa3c46bc4772960d: Status 404 returned error can't find the container with id c883fb82aaaa1e8d803aa0869c361c7b740411aeb33eb01faa3c46bc4772960d Oct 31 00:52:03.547333 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.547298 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq"] Oct 31 00:52:03.553448 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:52:03.553405 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c0ca28_c9be_4eea_8305_a87d12262907.slice/crio-69193dc6b85a96bf0d3979f5d26999adce4b567b736d0c085a5096819571e24e WatchSource:0}: Error finding container 69193dc6b85a96bf0d3979f5d26999adce4b567b736d0c085a5096819571e24e: Status 404 returned error can't find the container with id 69193dc6b85a96bf0d3979f5d26999adce4b567b736d0c085a5096819571e24e Oct 31 00:52:03.681557 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.681514 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" event={"ID":"b6c0ca28-c9be-4eea-8305-a87d12262907","Type":"ContainerStarted","Data":"a4696e2a61137d7b16015f80191e3a13587ba86e94e6318b5a28a562c2cd3620"} Oct 31 00:52:03.681791 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.681590 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" event={"ID":"b6c0ca28-c9be-4eea-8305-a87d12262907","Type":"ContainerStarted","Data":"69193dc6b85a96bf0d3979f5d26999adce4b567b736d0c085a5096819571e24e"} Oct 31 00:52:03.683299 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.683254 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" event={"ID":"86478869-cabb-4bd6-9204-3e6b34d189ab","Type":"ContainerStarted","Data":"ce5bfecbf2dc4c358c55ec8d716b22a5b03ed1fc1e516c865759536ca52ef0d3"} Oct 31 00:52:03.683426 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:03.683337 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" event={"ID":"86478869-cabb-4bd6-9204-3e6b34d189ab","Type":"ContainerStarted","Data":"c883fb82aaaa1e8d803aa0869c361c7b740411aeb33eb01faa3c46bc4772960d"} Oct 31 00:52:07.699031 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:07.698993 5821 generic.go:356] "Generic (PLEG): container finished" podID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerID="a4696e2a61137d7b16015f80191e3a13587ba86e94e6318b5a28a562c2cd3620" exitCode=0 Oct 31 00:52:07.699517 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:07.699070 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" event={"ID":"b6c0ca28-c9be-4eea-8305-a87d12262907","Type":"ContainerDied","Data":"a4696e2a61137d7b16015f80191e3a13587ba86e94e6318b5a28a562c2cd3620"} Oct 31 00:52:07.700783 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:07.700758 5821 generic.go:356] "Generic (PLEG): container finished" podID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerID="ce5bfecbf2dc4c358c55ec8d716b22a5b03ed1fc1e516c865759536ca52ef0d3" exitCode=0 Oct 31 00:52:07.700880 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:07.700801 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" event={"ID":"86478869-cabb-4bd6-9204-3e6b34d189ab","Type":"ContainerDied","Data":"ce5bfecbf2dc4c358c55ec8d716b22a5b03ed1fc1e516c865759536ca52ef0d3"} Oct 31 00:52:08.706503 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:08.706460 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" event={"ID":"b6c0ca28-c9be-4eea-8305-a87d12262907","Type":"ContainerStarted","Data":"5bd40ac11597d332597a29c7563b0146852f7f16e060732d10256418e78a39dd"} Oct 31 00:52:08.706953 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:08.706551 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" event={"ID":"b6c0ca28-c9be-4eea-8305-a87d12262907","Type":"ContainerStarted","Data":"b8b1aa2263ca33cf603476f21ef3c990956764c54ef8a7f5abeb19e363ae9b64"} Oct 31 00:52:08.706953 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:08.706774 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:08.708536 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:08.708511 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" event={"ID":"86478869-cabb-4bd6-9204-3e6b34d189ab","Type":"ContainerStarted","Data":"3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87"} Oct 31 00:52:08.708680 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:08.708558 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" event={"ID":"86478869-cabb-4bd6-9204-3e6b34d189ab","Type":"ContainerStarted","Data":"1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346"} Oct 31 00:52:08.708893 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:08.708875 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:08.708931 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:08.708924 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:08.710221 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:08.710199 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 31 00:52:08.726991 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:08.726953 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podStartSLOduration=2.89027707 podStartE2EDuration="6.726941211s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.836664141s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:08.725129492 +0000 UTC m=+1312.086712130" watchObservedRunningTime="2026-10-31 00:52:08.726941211 +0000 UTC m=+1312.088523846" Oct 31 00:52:08.743477 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:08.743431 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podStartSLOduration=2.97167387 podStartE2EDuration="6.743417928s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.771744058s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:08.742574603 +0000 UTC m=+1312.104157253" watchObservedRunningTime="2026-10-31 00:52:08.743417928 +0000 UTC m=+1312.105000566" Oct 31 00:52:09.712261 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:09.712220 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 31 00:52:09.712700 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:09.712332 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:09.713457 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:09.713426 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:52:10.715839 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:10.715809 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:52:14.716580 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:14.716552 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:52:14.717092 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:14.717064 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 31 00:52:15.721010 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:15.720984 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:52:15.721539 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:15.721514 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:52:19.837877 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:52:19.837841 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:52:24.717882 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:24.717841 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 31 00:52:25.721853 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:25.721811 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:52:34.717366 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:34.717318 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 31 00:52:35.722047 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:35.722012 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:52:44.717161 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:44.717119 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 31 00:52:45.721686 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:45.721647 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:52:49.838363 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:52:49.838326 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:52:54.717849 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:54.717812 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 31 00:52:55.722370 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:52:55.722337 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:53:04.717343 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:04.717264 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 31 00:53:05.722014 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:05.721978 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:53:14.717755 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:14.717720 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 31 00:53:15.721935 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:15.721900 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:53:19.773788 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:19.773740 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 31 00:53:19.838874 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:53:19.838841 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:53:25.721777 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:25.721749 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:53:29.774781 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:29.774745 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:53:33.158571 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:33.158537 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq"] Oct 31 00:53:33.159076 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:33.158874 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" containerID="cri-o://b8b1aa2263ca33cf603476f21ef3c990956764c54ef8a7f5abeb19e363ae9b64" gracePeriod=30 Oct 31 00:53:33.159252 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:33.159227 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kube-rbac-proxy" containerID="cri-o://5bd40ac11597d332597a29c7563b0146852f7f16e060732d10256418e78a39dd" gracePeriod=30 Oct 31 00:53:34.013287 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:34.013250 5821 generic.go:356] "Generic (PLEG): container finished" podID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerID="5bd40ac11597d332597a29c7563b0146852f7f16e060732d10256418e78a39dd" exitCode=0 Oct 31 00:53:34.013461 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:34.013324 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" event={"ID":"b6c0ca28-c9be-4eea-8305-a87d12262907","Type":"ContainerDied","Data":"5bd40ac11597d332597a29c7563b0146852f7f16e060732d10256418e78a39dd"} Oct 31 00:53:35.716871 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:35.716825 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused" Oct 31 00:53:35.722165 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:35.722138 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:53:38.032014 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.031982 5821 generic.go:356] "Generic (PLEG): container finished" podID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerID="b8b1aa2263ca33cf603476f21ef3c990956764c54ef8a7f5abeb19e363ae9b64" exitCode=0 Oct 31 00:53:38.032341 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.032049 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" event={"ID":"b6c0ca28-c9be-4eea-8305-a87d12262907","Type":"ContainerDied","Data":"b8b1aa2263ca33cf603476f21ef3c990956764c54ef8a7f5abeb19e363ae9b64"} Oct 31 00:53:38.106276 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.106253 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:53:38.185906 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.185876 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c0ca28-c9be-4eea-8305-a87d12262907-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6c0ca28-c9be-4eea-8305-a87d12262907-proxy-tls\") pod \"b6c0ca28-c9be-4eea-8305-a87d12262907\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " Oct 31 00:53:38.186063 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.185928 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-rbac-proxy-sar-config\") pod \"b6c0ca28-c9be-4eea-8305-a87d12262907\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " Oct 31 00:53:38.186063 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.185954 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-cabundle-cert\") pod \"b6c0ca28-c9be-4eea-8305-a87d12262907\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " Oct 31 00:53:38.186063 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.185982 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c0ca28-c9be-4eea-8305-a87d12262907-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c0ca28-c9be-4eea-8305-a87d12262907-kserve-provision-location\") pod \"b6c0ca28-c9be-4eea-8305-a87d12262907\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " Oct 31 00:53:38.186063 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.186014 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-api-access-nzjkj\" (UniqueName: \"kubernetes.io/projected/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-api-access-nzjkj\") pod \"b6c0ca28-c9be-4eea-8305-a87d12262907\" (UID: \"b6c0ca28-c9be-4eea-8305-a87d12262907\") " Oct 31 00:53:38.186347 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.186312 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c0ca28-c9be-4eea-8305-a87d12262907-kserve-provision-location" pod "b6c0ca28-c9be-4eea-8305-a87d12262907" (UID: "b6c0ca28-c9be-4eea-8305-a87d12262907"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:38.186455 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.186349 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-cabundle-cert" pod "b6c0ca28-c9be-4eea-8305-a87d12262907" (UID: "b6c0ca28-c9be-4eea-8305-a87d12262907"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:38.186455 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.186355 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-rbac-proxy-sar-config" pod "b6c0ca28-c9be-4eea-8305-a87d12262907" (UID: "b6c0ca28-c9be-4eea-8305-a87d12262907"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:38.188135 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.188110 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-api-access-nzjkj" pod "b6c0ca28-c9be-4eea-8305-a87d12262907" (UID: "b6c0ca28-c9be-4eea-8305-a87d12262907"). InnerVolumeSpecName "kube-api-access-nzjkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:38.188226 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.188186 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c0ca28-c9be-4eea-8305-a87d12262907-proxy-tls" pod "b6c0ca28-c9be-4eea-8305-a87d12262907" (UID: "b6c0ca28-c9be-4eea-8305-a87d12262907"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:38.287439 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.287415 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6c0ca28-c9be-4eea-8305-a87d12262907-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:38.287439 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.287437 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:38.287607 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.287448 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c0ca28-c9be-4eea-8305-a87d12262907-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:38.287607 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.287460 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c0ca28-c9be-4eea-8305-a87d12262907-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:38.287607 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:38.287469 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzjkj\" (UniqueName: \"kubernetes.io/projected/b6c0ca28-c9be-4eea-8305-a87d12262907-kube-api-access-nzjkj\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:39.037681 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:39.037595 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" event={"ID":"b6c0ca28-c9be-4eea-8305-a87d12262907","Type":"ContainerDied","Data":"69193dc6b85a96bf0d3979f5d26999adce4b567b736d0c085a5096819571e24e"} Oct 31 00:53:39.037681 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:39.037657 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq" Oct 31 00:53:39.037681 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:39.037677 5821 scope.go:122] "RemoveContainer" containerID="5bd40ac11597d332597a29c7563b0146852f7f16e060732d10256418e78a39dd" Oct 31 00:53:39.045605 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:39.045587 5821 scope.go:122] "RemoveContainer" containerID="b8b1aa2263ca33cf603476f21ef3c990956764c54ef8a7f5abeb19e363ae9b64" Oct 31 00:53:39.052592 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:39.052576 5821 scope.go:122] "RemoveContainer" containerID="a4696e2a61137d7b16015f80191e3a13587ba86e94e6318b5a28a562c2cd3620" Oct 31 00:53:39.055424 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:39.055395 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq"] Oct 31 00:53:39.059484 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:39.059463 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-promote-predictor-5cdb4c96cc-2hrwq"] Oct 31 00:53:40.777680 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:53:40.777649 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" path="/var/lib/kubelet/pods/b6c0ca28-c9be-4eea-8305-a87d12262907/volumes" Oct 31 00:53:49.839005 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:53:49.838969 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:54:19.837548 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:54:19.837512 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:54:49.840943 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:54:49.840891 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:55:03.585931 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:03.585880 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv"] Oct 31 00:55:03.586366 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:03.586170 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" containerID="cri-o://1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346" gracePeriod=30 Oct 31 00:55:03.586366 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:03.586171 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kube-rbac-proxy" containerID="cri-o://3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87" gracePeriod=30 Oct 31 00:55:04.334525 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:04.334491 5821 generic.go:356] "Generic (PLEG): container finished" podID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerID="3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87" exitCode=0 Oct 31 00:55:04.334710 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:04.334563 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" event={"ID":"86478869-cabb-4bd6-9204-3e6b34d189ab","Type":"ContainerDied","Data":"3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87"} Oct 31 00:55:04.713138 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:04.713094 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused" Oct 31 00:55:08.621823 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.621800 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:55:08.761722 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.761698 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-cabundle-cert\") pod \"86478869-cabb-4bd6-9204-3e6b34d189ab\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " Oct 31 00:55:08.761907 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.761730 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-rbac-proxy-sar-config\") pod \"86478869-cabb-4bd6-9204-3e6b34d189ab\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " Oct 31 00:55:08.761907 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.761752 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-api-access-8zfzd\" (UniqueName: \"kubernetes.io/projected/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-api-access-8zfzd\") pod \"86478869-cabb-4bd6-9204-3e6b34d189ab\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " Oct 31 00:55:08.761907 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.761788 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86478869-cabb-4bd6-9204-3e6b34d189ab-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86478869-cabb-4bd6-9204-3e6b34d189ab-kserve-provision-location\") pod \"86478869-cabb-4bd6-9204-3e6b34d189ab\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " Oct 31 00:55:08.761907 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.761822 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86478869-cabb-4bd6-9204-3e6b34d189ab-proxy-tls\" (UniqueName: \"kubernetes.io/secret/86478869-cabb-4bd6-9204-3e6b34d189ab-proxy-tls\") pod \"86478869-cabb-4bd6-9204-3e6b34d189ab\" (UID: \"86478869-cabb-4bd6-9204-3e6b34d189ab\") " Oct 31 00:55:08.762178 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.762151 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-rbac-proxy-sar-config" pod "86478869-cabb-4bd6-9204-3e6b34d189ab" (UID: "86478869-cabb-4bd6-9204-3e6b34d189ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:08.762178 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.762160 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-cabundle-cert" pod "86478869-cabb-4bd6-9204-3e6b34d189ab" (UID: "86478869-cabb-4bd6-9204-3e6b34d189ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:08.762262 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.762163 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86478869-cabb-4bd6-9204-3e6b34d189ab-kserve-provision-location" pod "86478869-cabb-4bd6-9204-3e6b34d189ab" (UID: "86478869-cabb-4bd6-9204-3e6b34d189ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:08.763988 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.763962 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-api-access-8zfzd" pod "86478869-cabb-4bd6-9204-3e6b34d189ab" (UID: "86478869-cabb-4bd6-9204-3e6b34d189ab"). InnerVolumeSpecName "kube-api-access-8zfzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:08.764122 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.764103 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86478869-cabb-4bd6-9204-3e6b34d189ab-proxy-tls" pod "86478869-cabb-4bd6-9204-3e6b34d189ab" (UID: "86478869-cabb-4bd6-9204-3e6b34d189ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:08.863102 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.863069 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:08.863102 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.863098 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:08.863102 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.863109 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zfzd\" (UniqueName: \"kubernetes.io/projected/86478869-cabb-4bd6-9204-3e6b34d189ab-kube-api-access-8zfzd\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:08.863315 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.863117 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86478869-cabb-4bd6-9204-3e6b34d189ab-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:08.863315 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:08.863127 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86478869-cabb-4bd6-9204-3e6b34d189ab-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:09.351836 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.351802 5821 generic.go:356] "Generic (PLEG): container finished" podID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerID="1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346" exitCode=0 Oct 31 00:55:09.352012 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.351871 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" Oct 31 00:55:09.352012 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.351894 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" event={"ID":"86478869-cabb-4bd6-9204-3e6b34d189ab","Type":"ContainerDied","Data":"1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346"} Oct 31 00:55:09.352012 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.351961 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv" event={"ID":"86478869-cabb-4bd6-9204-3e6b34d189ab","Type":"ContainerDied","Data":"c883fb82aaaa1e8d803aa0869c361c7b740411aeb33eb01faa3c46bc4772960d"} Oct 31 00:55:09.352012 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.351988 5821 scope.go:122] "RemoveContainer" containerID="3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87" Oct 31 00:55:09.360325 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.360308 5821 scope.go:122] "RemoveContainer" containerID="1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346" Oct 31 00:55:09.367561 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.367545 5821 scope.go:122] "RemoveContainer" containerID="ce5bfecbf2dc4c358c55ec8d716b22a5b03ed1fc1e516c865759536ca52ef0d3" Oct 31 00:55:09.370274 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.370255 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv"] Oct 31 00:55:09.375302 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.375281 5821 scope.go:122] "RemoveContainer" containerID="3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87" Oct 31 00:55:09.375595 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:55:09.375574 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87\": container with ID starting with 3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87 not found: ID does not exist" containerID="3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87" Oct 31 00:55:09.376110 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.375602 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87"} err="failed to get container status \"3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87\": rpc error: code = NotFound desc = could not find container \"3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87\": container with ID starting with 3f2c2d866d16f50a25116e8d51a028ba133db1a1172204581d46eb235acf8d87 not found: ID does not exist" Oct 31 00:55:09.376110 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.375645 5821 scope.go:122] "RemoveContainer" containerID="1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346" Oct 31 00:55:09.376110 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.375732 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-c950106202/isvc-canary-promote-v2-predictor-5997987886-lqjsv"] Oct 31 00:55:09.376110 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:55:09.375926 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346\": container with ID starting with 1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346 not found: ID does not exist" containerID="1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346" Oct 31 00:55:09.376110 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.375946 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346"} err="failed to get container status \"1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346\": rpc error: code = NotFound desc = could not find container \"1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346\": container with ID starting with 1478782e893489e0b1350c71f58fb822c3ae22bcc8a2e5826abea3820a096346 not found: ID does not exist" Oct 31 00:55:09.376110 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.375960 5821 scope.go:122] "RemoveContainer" containerID="ce5bfecbf2dc4c358c55ec8d716b22a5b03ed1fc1e516c865759536ca52ef0d3" Oct 31 00:55:09.376335 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:55:09.376170 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5bfecbf2dc4c358c55ec8d716b22a5b03ed1fc1e516c865759536ca52ef0d3\": container with ID starting with ce5bfecbf2dc4c358c55ec8d716b22a5b03ed1fc1e516c865759536ca52ef0d3 not found: ID does not exist" containerID="ce5bfecbf2dc4c358c55ec8d716b22a5b03ed1fc1e516c865759536ca52ef0d3" Oct 31 00:55:09.376335 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:09.376189 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5bfecbf2dc4c358c55ec8d716b22a5b03ed1fc1e516c865759536ca52ef0d3"} err="failed to get container status \"ce5bfecbf2dc4c358c55ec8d716b22a5b03ed1fc1e516c865759536ca52ef0d3\": rpc error: code = NotFound desc = could not find container \"ce5bfecbf2dc4c358c55ec8d716b22a5b03ed1fc1e516c865759536ca52ef0d3\": container with ID starting with ce5bfecbf2dc4c358c55ec8d716b22a5b03ed1fc1e516c865759536ca52ef0d3 not found: ID does not exist" Oct 31 00:55:10.777013 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:10.776980 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" path="/var/lib/kubelet/pods/86478869-cabb-4bd6-9204-3e6b34d189ab/volumes" Oct 31 00:55:19.837270 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:55:19.837232 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:55:21.919304 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919271 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9"] Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919600 5821 cpu_manager.go:414] "containerMap: removing container" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kube-rbac-proxy" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919614 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kube-rbac-proxy" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919646 5821 cpu_manager.go:414] "containerMap: removing container" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kube-rbac-proxy" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919654 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kube-rbac-proxy" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919672 5821 cpu_manager.go:414] "containerMap: removing container" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="storage-initializer" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919677 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="storage-initializer" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919686 5821 cpu_manager.go:414] "containerMap: removing container" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919691 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919700 5821 cpu_manager.go:414] "containerMap: removing container" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919705 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919713 5821 cpu_manager.go:414] "containerMap: removing container" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="storage-initializer" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919717 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="storage-initializer" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919759 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kube-rbac-proxy" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919769 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kserve-container" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919775 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="86478869-cabb-4bd6-9204-3e6b34d189ab" containerName="kserve-container" Oct 31 00:55:21.919941 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.919783 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c0ca28-c9be-4eea-8305-a87d12262907" containerName="kube-rbac-proxy" Oct 31 00:55:21.924115 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.924100 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:21.931818 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.931798 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4ae23484d1\"/\"kube-root-ca.crt\"" Oct 31 00:55:21.931958 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.931839 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4ae23484d1\"/\"openshift-service-ca.crt\"" Oct 31 00:55:21.932041 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.931932 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4ae23484d1\"/\"default-dockercfg-7r2rx\"" Oct 31 00:55:21.932120 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.932054 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4ae23484d1\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\"" Oct 31 00:55:21.932120 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.932105 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4ae23484d1\"/\"message-dumper-raw-cipn-predictor-serving-cert\"" Oct 31 00:55:21.932664 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:21.932643 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9"] Oct 31 00:55:22.065363 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:22.065328 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fead87f-c6a9-4251-bac4-914abd70ee02-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-6956785bd-rtml9\" (UID: \"9fead87f-c6a9-4251-bac4-914abd70ee02\") " pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:22.065574 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:22.065376 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5ptz\" (UniqueName: \"kubernetes.io/projected/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-api-access-z5ptz\") pod \"message-dumper-raw-cipn-predictor-6956785bd-rtml9\" (UID: \"9fead87f-c6a9-4251-bac4-914abd70ee02\") " pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:22.065574 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:22.065472 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-6956785bd-rtml9\" (UID: \"9fead87f-c6a9-4251-bac4-914abd70ee02\") " pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:22.165934 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:22.165900 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fead87f-c6a9-4251-bac4-914abd70ee02-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-6956785bd-rtml9\" (UID: \"9fead87f-c6a9-4251-bac4-914abd70ee02\") " pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:22.165934 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:22.165939 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5ptz\" (UniqueName: \"kubernetes.io/projected/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-api-access-z5ptz\") pod \"message-dumper-raw-cipn-predictor-6956785bd-rtml9\" (UID: \"9fead87f-c6a9-4251-bac4-914abd70ee02\") " pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:22.166187 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:22.165981 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-6956785bd-rtml9\" (UID: \"9fead87f-c6a9-4251-bac4-914abd70ee02\") " pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:22.166594 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:22.166574 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-6956785bd-rtml9\" (UID: \"9fead87f-c6a9-4251-bac4-914abd70ee02\") " pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:22.168393 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:22.168374 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fead87f-c6a9-4251-bac4-914abd70ee02-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-6956785bd-rtml9\" (UID: \"9fead87f-c6a9-4251-bac4-914abd70ee02\") " pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:22.174347 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:22.174293 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5ptz\" (UniqueName: \"kubernetes.io/projected/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-api-access-z5ptz\") pod \"message-dumper-raw-cipn-predictor-6956785bd-rtml9\" (UID: \"9fead87f-c6a9-4251-bac4-914abd70ee02\") " pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:22.240123 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:22.240099 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:22.512175 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:22.512099 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9"] Oct 31 00:55:22.515091 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:55:22.515066 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fead87f_c6a9_4251_bac4_914abd70ee02.slice/crio-c79ecd3caf87c9c80dc690077cac1f9e73607ae4093ddc131e3bc9666ae203ec WatchSource:0}: Error finding container c79ecd3caf87c9c80dc690077cac1f9e73607ae4093ddc131e3bc9666ae203ec: Status 404 returned error can't find the container with id c79ecd3caf87c9c80dc690077cac1f9e73607ae4093ddc131e3bc9666ae203ec Oct 31 00:55:23.401543 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:23.401504 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" event={"ID":"9fead87f-c6a9-4251-bac4-914abd70ee02","Type":"ContainerStarted","Data":"d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e"} Oct 31 00:55:23.402060 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:23.401577 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" event={"ID":"9fead87f-c6a9-4251-bac4-914abd70ee02","Type":"ContainerStarted","Data":"ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba"} Oct 31 00:55:23.402060 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:23.401605 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" event={"ID":"9fead87f-c6a9-4251-bac4-914abd70ee02","Type":"ContainerStarted","Data":"c79ecd3caf87c9c80dc690077cac1f9e73607ae4093ddc131e3bc9666ae203ec"} Oct 31 00:55:23.402060 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:23.401796 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:23.402060 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:23.401852 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:23.403587 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:23.403568 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:23.420477 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:23.420434 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" podStartSLOduration=2.420423358 podStartE2EDuration="2.420423358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:23.419940523 +0000 UTC m=+1506.781523161" watchObservedRunningTime="2026-10-31 00:55:23.420423358 +0000 UTC m=+1506.782005992" Oct 31 00:55:30.414164 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:30.414136 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:55:32.021361 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.021322 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx"] Oct 31 00:55:32.025111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.025088 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.029329 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.029304 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4ae23484d1\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:55:32.029329 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.029304 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4ae23484d1\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\"" Oct 31 00:55:32.029475 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.029328 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4ae23484d1\"/\"seaweedfs-s3-creds\"" Oct 31 00:55:32.029827 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.029808 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4ae23484d1\"/\"isvc-logger-raw-cipn-predictor-serving-cert\"" Oct 31 00:55:32.035844 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.035822 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx"] Oct 31 00:55:32.144133 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.144101 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.144299 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.144142 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxzpw\" (UniqueName: \"kubernetes.io/projected/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-api-access-nxzpw\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.144299 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.144253 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0331cc6e-cbd1-418c-be48-6d168b66c30d-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.144299 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.144294 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0331cc6e-cbd1-418c-be48-6d168b66c30d-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.144423 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.144384 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.244972 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.244940 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.244972 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.244976 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzpw\" (UniqueName: \"kubernetes.io/projected/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-api-access-nxzpw\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.245190 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.245006 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0331cc6e-cbd1-418c-be48-6d168b66c30d-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.245190 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.245039 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0331cc6e-cbd1-418c-be48-6d168b66c30d-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.245190 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.245073 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.245500 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.245478 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0331cc6e-cbd1-418c-be48-6d168b66c30d-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.245642 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.245602 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.245732 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.245713 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.247626 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.247600 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0331cc6e-cbd1-418c-be48-6d168b66c30d-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.254362 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.254337 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxzpw\" (UniqueName: \"kubernetes.io/projected/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-api-access-nxzpw\") pod \"isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.346851 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.346765 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:32.621862 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:32.621706 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx"] Oct 31 00:55:32.625054 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:55:32.625025 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0331cc6e_cbd1_418c_be48_6d168b66c30d.slice/crio-46a2f12d8e75877024b742186242dacd74b27bea1a697b3e8167c5e00a139da8 WatchSource:0}: Error finding container 46a2f12d8e75877024b742186242dacd74b27bea1a697b3e8167c5e00a139da8: Status 404 returned error can't find the container with id 46a2f12d8e75877024b742186242dacd74b27bea1a697b3e8167c5e00a139da8 Oct 31 00:55:33.435726 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:33.435688 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" event={"ID":"0331cc6e-cbd1-418c-be48-6d168b66c30d","Type":"ContainerStarted","Data":"29a3cc48dad5c3dcbe10bd533e5765b8d7a895406e680ba6a932dc4402ea80e7"} Oct 31 00:55:33.436209 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:33.435778 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" event={"ID":"0331cc6e-cbd1-418c-be48-6d168b66c30d","Type":"ContainerStarted","Data":"46a2f12d8e75877024b742186242dacd74b27bea1a697b3e8167c5e00a139da8"} Oct 31 00:55:37.450854 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:37.450819 5821 generic.go:356] "Generic (PLEG): container finished" podID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerID="29a3cc48dad5c3dcbe10bd533e5765b8d7a895406e680ba6a932dc4402ea80e7" exitCode=0 Oct 31 00:55:37.451257 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:37.450891 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" event={"ID":"0331cc6e-cbd1-418c-be48-6d168b66c30d","Type":"ContainerDied","Data":"29a3cc48dad5c3dcbe10bd533e5765b8d7a895406e680ba6a932dc4402ea80e7"} Oct 31 00:55:38.457150 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:38.457119 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" event={"ID":"0331cc6e-cbd1-418c-be48-6d168b66c30d","Type":"ContainerStarted","Data":"a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621"} Oct 31 00:55:38.457658 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:38.457191 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" event={"ID":"0331cc6e-cbd1-418c-be48-6d168b66c30d","Type":"ContainerStarted","Data":"6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff"} Oct 31 00:55:38.457658 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:38.457215 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" event={"ID":"0331cc6e-cbd1-418c-be48-6d168b66c30d","Type":"ContainerStarted","Data":"d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0"} Oct 31 00:55:38.457658 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:38.457634 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:38.457795 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:38.457687 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:38.457795 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:38.457716 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:38.458927 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:38.458902 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:55:38.459573 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:38.459551 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:55:38.485194 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:38.485131 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podStartSLOduration=2.506285722 podStartE2EDuration="6.485112598s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.978826876s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:38.483242512 +0000 UTC m=+1521.844825147" watchObservedRunningTime="2026-10-31 00:55:38.485112598 +0000 UTC m=+1521.846695243" Oct 31 00:55:39.461017 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:39.460974 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:55:39.461447 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:39.461374 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:55:44.464914 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:44.464886 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:55:44.465465 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:44.465442 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:55:44.465713 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:44.465690 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:55:49.837836 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:55:49.837797 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:55:54.465369 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:54.465314 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:55:54.465917 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:55:54.465832 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:04.465539 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:04.465451 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:56:04.465998 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:04.465844 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:14.466182 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:14.466137 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:56:14.466592 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:14.466561 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:19.838681 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:56:19.838645 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:56:24.465728 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:24.465692 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:56:24.466207 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:24.466184 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:34.466232 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:34.466173 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:56:34.466803 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:34.466713 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:44.465921 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:44.465875 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:56:44.466358 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:44.466276 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:45.773204 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:45.773157 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:56:45.773651 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:45.773435 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:49.838660 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:56:49.838599 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:56:55.774609 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:55.774581 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:56:55.775038 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:56:55.774715 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:57:07.727479 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:07.727440 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-4ae23484d1_message-dumper-raw-cipn-predictor-6956785bd-rtml9_9fead87f-c6a9-4251-bac4-914abd70ee02/kserve-container/0.log" Oct 31 00:57:08.039506 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.039421 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx"] Oct 31 00:57:08.039753 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.039706 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" containerID="cri-o://d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0" gracePeriod=30 Oct 31 00:57:08.039753 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.039738 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kube-rbac-proxy" containerID="cri-o://6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff" gracePeriod=30 Oct 31 00:57:08.039998 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.039742 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" containerID="cri-o://a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621" gracePeriod=30 Oct 31 00:57:08.068768 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.068743 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9"] Oct 31 00:57:08.068977 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.068945 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" podUID="9fead87f-c6a9-4251-bac4-914abd70ee02" containerName="kserve-container" containerID="cri-o://ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba" gracePeriod=30 Oct 31 00:57:08.069069 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.068982 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" podUID="9fead87f-c6a9-4251-bac4-914abd70ee02" containerName="kube-rbac-proxy" containerID="cri-o://d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e" gracePeriod=30 Oct 31 00:57:08.301317 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.301261 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:57:08.390905 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.390869 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fead87f-c6a9-4251-bac4-914abd70ee02-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fead87f-c6a9-4251-bac4-914abd70ee02-proxy-tls\") pod \"9fead87f-c6a9-4251-bac4-914abd70ee02\" (UID: \"9fead87f-c6a9-4251-bac4-914abd70ee02\") " Oct 31 00:57:08.391101 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.390922 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-api-access-z5ptz\" (UniqueName: \"kubernetes.io/projected/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-api-access-z5ptz\") pod \"9fead87f-c6a9-4251-bac4-914abd70ee02\" (UID: \"9fead87f-c6a9-4251-bac4-914abd70ee02\") " Oct 31 00:57:08.391101 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.390964 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-rbac-proxy-sar-config\") pod \"9fead87f-c6a9-4251-bac4-914abd70ee02\" (UID: \"9fead87f-c6a9-4251-bac4-914abd70ee02\") " Oct 31 00:57:08.391374 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.391349 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-rbac-proxy-sar-config" pod "9fead87f-c6a9-4251-bac4-914abd70ee02" (UID: "9fead87f-c6a9-4251-bac4-914abd70ee02"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:08.393327 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.393306 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-api-access-z5ptz" pod "9fead87f-c6a9-4251-bac4-914abd70ee02" (UID: "9fead87f-c6a9-4251-bac4-914abd70ee02"). InnerVolumeSpecName "kube-api-access-z5ptz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:08.393395 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.393322 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fead87f-c6a9-4251-bac4-914abd70ee02-proxy-tls" pod "9fead87f-c6a9-4251-bac4-914abd70ee02" (UID: "9fead87f-c6a9-4251-bac4-914abd70ee02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:08.491903 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.491861 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fead87f-c6a9-4251-bac4-914abd70ee02-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:08.491903 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.491899 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5ptz\" (UniqueName: \"kubernetes.io/projected/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-api-access-z5ptz\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:08.492071 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.491913 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fead87f-c6a9-4251-bac4-914abd70ee02-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:08.778186 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.778151 5821 generic.go:356] "Generic (PLEG): container finished" podID="9fead87f-c6a9-4251-bac4-914abd70ee02" containerID="d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e" exitCode=0 Oct 31 00:57:08.778186 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.778178 5821 generic.go:356] "Generic (PLEG): container finished" podID="9fead87f-c6a9-4251-bac4-914abd70ee02" containerID="ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba" exitCode=2 Oct 31 00:57:08.778711 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.778275 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" Oct 31 00:57:08.779846 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.779821 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" event={"ID":"9fead87f-c6a9-4251-bac4-914abd70ee02","Type":"ContainerDied","Data":"d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e"} Oct 31 00:57:08.779986 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.779881 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" event={"ID":"9fead87f-c6a9-4251-bac4-914abd70ee02","Type":"ContainerDied","Data":"ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba"} Oct 31 00:57:08.779986 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.779913 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9" event={"ID":"9fead87f-c6a9-4251-bac4-914abd70ee02","Type":"ContainerDied","Data":"c79ecd3caf87c9c80dc690077cac1f9e73607ae4093ddc131e3bc9666ae203ec"} Oct 31 00:57:08.779986 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.779946 5821 scope.go:122] "RemoveContainer" containerID="d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e" Oct 31 00:57:08.781109 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.781090 5821 generic.go:356] "Generic (PLEG): container finished" podID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerID="6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff" exitCode=0 Oct 31 00:57:08.781196 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.781130 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" event={"ID":"0331cc6e-cbd1-418c-be48-6d168b66c30d","Type":"ContainerDied","Data":"6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff"} Oct 31 00:57:08.789151 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.789128 5821 scope.go:122] "RemoveContainer" containerID="ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba" Oct 31 00:57:08.797935 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.797920 5821 scope.go:122] "RemoveContainer" containerID="d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e" Oct 31 00:57:08.798226 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:57:08.798201 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e\": container with ID starting with d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e not found: ID does not exist" containerID="d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e" Oct 31 00:57:08.798280 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.798238 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e"} err="failed to get container status \"d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e\": rpc error: code = NotFound desc = could not find container \"d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e\": container with ID starting with d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e not found: ID does not exist" Oct 31 00:57:08.798280 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.798263 5821 scope.go:122] "RemoveContainer" containerID="ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba" Oct 31 00:57:08.798536 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:57:08.798517 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba\": container with ID starting with ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba not found: ID does not exist" containerID="ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba" Oct 31 00:57:08.798574 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.798544 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba"} err="failed to get container status \"ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba\": rpc error: code = NotFound desc = could not find container \"ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba\": container with ID starting with ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba not found: ID does not exist" Oct 31 00:57:08.798574 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.798563 5821 scope.go:122] "RemoveContainer" containerID="d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e" Oct 31 00:57:08.798849 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.798829 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e"} err="failed to get container status \"d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e\": rpc error: code = NotFound desc = could not find container \"d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e\": container with ID starting with d6302cfe0cdb1a22256a23e86f128f7e538f5b8c80cc0f62f0e693447acecc1e not found: ID does not exist" Oct 31 00:57:08.798897 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.798850 5821 scope.go:122] "RemoveContainer" containerID="ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba" Oct 31 00:57:08.799073 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.799059 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba"} err="failed to get container status \"ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba\": rpc error: code = NotFound desc = could not find container \"ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba\": container with ID starting with ab9da5945ad68828c2911a00fc04ca4a45a56b562b9abe44f8b4cdc4607f51ba not found: ID does not exist" Oct 31 00:57:08.804416 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.804397 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9"] Oct 31 00:57:08.810882 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:08.810863 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4ae23484d1/message-dumper-raw-cipn-predictor-6956785bd-rtml9"] Oct 31 00:57:09.461506 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:09.461469 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused" Oct 31 00:57:10.777611 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:10.777571 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fead87f-c6a9-4251-bac4-914abd70ee02" path="/var/lib/kubelet/pods/9fead87f-c6a9-4251-bac4-914abd70ee02/volumes" Oct 31 00:57:13.799831 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:13.799797 5821 generic.go:356] "Generic (PLEG): container finished" podID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerID="d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0" exitCode=0 Oct 31 00:57:13.800228 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:13.799865 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" event={"ID":"0331cc6e-cbd1-418c-be48-6d168b66c30d","Type":"ContainerDied","Data":"d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0"} Oct 31 00:57:14.461531 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:14.461494 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused" Oct 31 00:57:15.773156 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:15.773110 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:57:15.773648 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:15.773451 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:57:19.461260 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:19.461217 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused" Oct 31 00:57:19.461681 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:19.461344 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:57:19.838056 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:57:19.837977 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:57:24.461351 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:24.461309 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused" Oct 31 00:57:25.773608 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:25.773558 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:57:25.774115 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:25.773936 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:57:29.462078 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:29.461992 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused" Oct 31 00:57:34.461176 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:34.461131 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused" Oct 31 00:57:35.773135 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:35.773091 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:57:35.773583 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:35.773235 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:57:35.773583 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:35.773487 5821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:57:35.773583 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:35.773579 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:57:38.182695 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.182672 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:57:38.308252 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.308171 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-rbac-proxy-sar-config\") pod \"0331cc6e-cbd1-418c-be48-6d168b66c30d\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " Oct 31 00:57:38.308252 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.308217 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-cabundle-cert\") pod \"0331cc6e-cbd1-418c-be48-6d168b66c30d\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " Oct 31 00:57:38.308252 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.308248 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0331cc6e-cbd1-418c-be48-6d168b66c30d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0331cc6e-cbd1-418c-be48-6d168b66c30d-kserve-provision-location\") pod \"0331cc6e-cbd1-418c-be48-6d168b66c30d\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " Oct 31 00:57:38.308549 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.308276 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0331cc6e-cbd1-418c-be48-6d168b66c30d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/0331cc6e-cbd1-418c-be48-6d168b66c30d-proxy-tls\") pod \"0331cc6e-cbd1-418c-be48-6d168b66c30d\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " Oct 31 00:57:38.308549 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.308351 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-api-access-nxzpw\" (UniqueName: \"kubernetes.io/projected/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-api-access-nxzpw\") pod \"0331cc6e-cbd1-418c-be48-6d168b66c30d\" (UID: \"0331cc6e-cbd1-418c-be48-6d168b66c30d\") " Oct 31 00:57:38.308672 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.308555 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0331cc6e-cbd1-418c-be48-6d168b66c30d-kserve-provision-location" pod "0331cc6e-cbd1-418c-be48-6d168b66c30d" (UID: "0331cc6e-cbd1-418c-be48-6d168b66c30d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:38.308672 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.308651 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-rbac-proxy-sar-config" pod "0331cc6e-cbd1-418c-be48-6d168b66c30d" (UID: "0331cc6e-cbd1-418c-be48-6d168b66c30d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:38.308749 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.308664 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-cabundle-cert" pod "0331cc6e-cbd1-418c-be48-6d168b66c30d" (UID: "0331cc6e-cbd1-418c-be48-6d168b66c30d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:38.310576 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.310544 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-api-access-nxzpw" pod "0331cc6e-cbd1-418c-be48-6d168b66c30d" (UID: "0331cc6e-cbd1-418c-be48-6d168b66c30d"). InnerVolumeSpecName "kube-api-access-nxzpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:38.310576 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.310551 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0331cc6e-cbd1-418c-be48-6d168b66c30d-proxy-tls" pod "0331cc6e-cbd1-418c-be48-6d168b66c30d" (UID: "0331cc6e-cbd1-418c-be48-6d168b66c30d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:38.409604 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.409559 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxzpw\" (UniqueName: \"kubernetes.io/projected/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-api-access-nxzpw\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:38.409604 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.409599 5821 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:38.409604 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.409610 5821 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0331cc6e-cbd1-418c-be48-6d168b66c30d-cabundle-cert\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:38.409852 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.409641 5821 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0331cc6e-cbd1-418c-be48-6d168b66c30d-kserve-provision-location\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:38.409852 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.409651 5821 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0331cc6e-cbd1-418c-be48-6d168b66c30d-proxy-tls\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:38.882461 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.882423 5821 generic.go:356] "Generic (PLEG): container finished" podID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerID="a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621" exitCode=0 Oct 31 00:57:38.882748 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.882466 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" event={"ID":"0331cc6e-cbd1-418c-be48-6d168b66c30d","Type":"ContainerDied","Data":"a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621"} Oct 31 00:57:38.882748 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.882526 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" Oct 31 00:57:38.882748 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.882559 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx" event={"ID":"0331cc6e-cbd1-418c-be48-6d168b66c30d","Type":"ContainerDied","Data":"46a2f12d8e75877024b742186242dacd74b27bea1a697b3e8167c5e00a139da8"} Oct 31 00:57:38.882748 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.882589 5821 scope.go:122] "RemoveContainer" containerID="a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621" Oct 31 00:57:38.890878 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.890860 5821 scope.go:122] "RemoveContainer" containerID="6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff" Oct 31 00:57:38.897998 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.897983 5821 scope.go:122] "RemoveContainer" containerID="d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0" Oct 31 00:57:38.901965 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.901890 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx"] Oct 31 00:57:38.905752 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.905732 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4ae23484d1/isvc-logger-raw-cipn-predictor-598cfc5d77-c8btx"] Oct 31 00:57:38.906442 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.906421 5821 scope.go:122] "RemoveContainer" containerID="29a3cc48dad5c3dcbe10bd533e5765b8d7a895406e680ba6a932dc4402ea80e7" Oct 31 00:57:38.913463 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.913443 5821 scope.go:122] "RemoveContainer" containerID="a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621" Oct 31 00:57:38.913734 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:57:38.913715 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621\": container with ID starting with a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621 not found: ID does not exist" containerID="a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621" Oct 31 00:57:38.913818 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.913743 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621"} err="failed to get container status \"a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621\": rpc error: code = NotFound desc = could not find container \"a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621\": container with ID starting with a0462ba257c4481088534e20c396d214d431049553082e324d0b092f002a0621 not found: ID does not exist" Oct 31 00:57:38.913818 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.913762 5821 scope.go:122] "RemoveContainer" containerID="6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff" Oct 31 00:57:38.914029 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:57:38.914015 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff\": container with ID starting with 6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff not found: ID does not exist" containerID="6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff" Oct 31 00:57:38.914070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.914035 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff"} err="failed to get container status \"6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff\": rpc error: code = NotFound desc = could not find container \"6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff\": container with ID starting with 6fa71f19a81abafe01f3c443158ecc90601ebd5af952a772860b4ac8ea1cc7ff not found: ID does not exist" Oct 31 00:57:38.914070 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.914048 5821 scope.go:122] "RemoveContainer" containerID="d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0" Oct 31 00:57:38.914267 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:57:38.914249 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0\": container with ID starting with d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0 not found: ID does not exist" containerID="d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0" Oct 31 00:57:38.914309 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.914271 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0"} err="failed to get container status \"d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0\": rpc error: code = NotFound desc = could not find container \"d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0\": container with ID starting with d4a6612e4bc57884373a37b5ee6c444eef8a96f66a13c4f2b5c791bb4276f7b0 not found: ID does not exist" Oct 31 00:57:38.914309 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.914284 5821 scope.go:122] "RemoveContainer" containerID="29a3cc48dad5c3dcbe10bd533e5765b8d7a895406e680ba6a932dc4402ea80e7" Oct 31 00:57:38.914478 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:57:38.914462 5821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a3cc48dad5c3dcbe10bd533e5765b8d7a895406e680ba6a932dc4402ea80e7\": container with ID starting with 29a3cc48dad5c3dcbe10bd533e5765b8d7a895406e680ba6a932dc4402ea80e7 not found: ID does not exist" containerID="29a3cc48dad5c3dcbe10bd533e5765b8d7a895406e680ba6a932dc4402ea80e7" Oct 31 00:57:38.914517 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:38.914482 5821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a3cc48dad5c3dcbe10bd533e5765b8d7a895406e680ba6a932dc4402ea80e7"} err="failed to get container status \"29a3cc48dad5c3dcbe10bd533e5765b8d7a895406e680ba6a932dc4402ea80e7\": rpc error: code = NotFound desc = could not find container \"29a3cc48dad5c3dcbe10bd533e5765b8d7a895406e680ba6a932dc4402ea80e7\": container with ID starting with 29a3cc48dad5c3dcbe10bd533e5765b8d7a895406e680ba6a932dc4402ea80e7 not found: ID does not exist" Oct 31 00:57:40.778135 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:40.778086 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" path="/var/lib/kubelet/pods/0331cc6e-cbd1-418c-be48-6d168b66c30d/volumes" Oct 31 00:57:41.337478 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337446 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zpqhs/must-gather-62vpf"] Oct 31 00:57:41.337747 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337734 5821 cpu_manager.go:414] "containerMap: removing container" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kube-rbac-proxy" Oct 31 00:57:41.337799 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337749 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kube-rbac-proxy" Oct 31 00:57:41.337799 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337761 5821 cpu_manager.go:414] "containerMap: removing container" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" Oct 31 00:57:41.337799 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337766 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" Oct 31 00:57:41.337799 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337775 5821 cpu_manager.go:414] "containerMap: removing container" podUID="9fead87f-c6a9-4251-bac4-914abd70ee02" containerName="kserve-container" Oct 31 00:57:41.337799 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337780 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fead87f-c6a9-4251-bac4-914abd70ee02" containerName="kserve-container" Oct 31 00:57:41.337799 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337787 5821 cpu_manager.go:414] "containerMap: removing container" podUID="9fead87f-c6a9-4251-bac4-914abd70ee02" containerName="kube-rbac-proxy" Oct 31 00:57:41.337799 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337792 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fead87f-c6a9-4251-bac4-914abd70ee02" containerName="kube-rbac-proxy" Oct 31 00:57:41.337799 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337802 5821 cpu_manager.go:414] "containerMap: removing container" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" Oct 31 00:57:41.338039 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337806 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" Oct 31 00:57:41.338039 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337821 5821 cpu_manager.go:414] "containerMap: removing container" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="storage-initializer" Oct 31 00:57:41.338039 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337826 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="storage-initializer" Oct 31 00:57:41.338039 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337878 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kserve-container" Oct 31 00:57:41.338039 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337888 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="kube-rbac-proxy" Oct 31 00:57:41.338039 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337894 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="0331cc6e-cbd1-418c-be48-6d168b66c30d" containerName="agent" Oct 31 00:57:41.338039 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337900 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fead87f-c6a9-4251-bac4-914abd70ee02" containerName="kserve-container" Oct 31 00:57:41.338039 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.337905 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fead87f-c6a9-4251-bac4-914abd70ee02" containerName="kube-rbac-proxy" Oct 31 00:57:41.342033 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.342016 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:57:41.349463 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.349437 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zpqhs\"/\"default-dockercfg-6xfwr\"" Oct 31 00:57:41.350165 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.349902 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpqhs\"/\"kube-root-ca.crt\"" Oct 31 00:57:41.350165 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.350136 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpqhs\"/\"openshift-service-ca.crt\"" Oct 31 00:57:41.350505 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.350489 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpqhs/must-gather-62vpf"] Oct 31 00:57:41.434579 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.434545 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55gmm\" (UniqueName: \"kubernetes.io/projected/226286a2-cfc1-46a8-8383-41b449260d51-kube-api-access-55gmm\") pod \"must-gather-62vpf\" (UID: \"226286a2-cfc1-46a8-8383-41b449260d51\") " pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:57:41.434753 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.434597 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/226286a2-cfc1-46a8-8383-41b449260d51-must-gather-output\") pod \"must-gather-62vpf\" (UID: \"226286a2-cfc1-46a8-8383-41b449260d51\") " pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:57:41.535753 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.535698 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55gmm\" (UniqueName: \"kubernetes.io/projected/226286a2-cfc1-46a8-8383-41b449260d51-kube-api-access-55gmm\") pod \"must-gather-62vpf\" (UID: \"226286a2-cfc1-46a8-8383-41b449260d51\") " pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:57:41.535921 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.535766 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/226286a2-cfc1-46a8-8383-41b449260d51-must-gather-output\") pod \"must-gather-62vpf\" (UID: \"226286a2-cfc1-46a8-8383-41b449260d51\") " pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:57:41.536057 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.536036 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/226286a2-cfc1-46a8-8383-41b449260d51-must-gather-output\") pod \"must-gather-62vpf\" (UID: \"226286a2-cfc1-46a8-8383-41b449260d51\") " pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:57:41.545428 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.545402 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55gmm\" (UniqueName: \"kubernetes.io/projected/226286a2-cfc1-46a8-8383-41b449260d51-kube-api-access-55gmm\") pod \"must-gather-62vpf\" (UID: \"226286a2-cfc1-46a8-8383-41b449260d51\") " pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:57:41.662803 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.662776 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:57:41.933889 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.933805 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpqhs/must-gather-62vpf"] Oct 31 00:57:41.936898 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:57:41.936873 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod226286a2_cfc1_46a8_8383_41b449260d51.slice/crio-b8f6c113228ae01f2f8bf0656a35de8394bd65803df5eda5989729e8e5f0ebe0 WatchSource:0}: Error finding container b8f6c113228ae01f2f8bf0656a35de8394bd65803df5eda5989729e8e5f0ebe0: Status 404 returned error can't find the container with id b8f6c113228ae01f2f8bf0656a35de8394bd65803df5eda5989729e8e5f0ebe0 Oct 31 00:57:41.939148 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:41.939132 5821 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:57:42.903763 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:42.903727 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpqhs/must-gather-62vpf" event={"ID":"226286a2-cfc1-46a8-8383-41b449260d51","Type":"ContainerStarted","Data":"b8f6c113228ae01f2f8bf0656a35de8394bd65803df5eda5989729e8e5f0ebe0"} Oct 31 00:57:48.934404 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:48.934366 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpqhs/must-gather-62vpf" event={"ID":"226286a2-cfc1-46a8-8383-41b449260d51","Type":"ContainerStarted","Data":"e75d383acd848a479ddba8b9994c3eb6930be3d3f231ae6868ac5a47d81c380f"} Oct 31 00:57:48.934878 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:48.934423 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpqhs/must-gather-62vpf" event={"ID":"226286a2-cfc1-46a8-8383-41b449260d51","Type":"ContainerStarted","Data":"f2933c508f74a422aaf1901764ecd8cffc85b6cb76b12d1dc668dc3d9423dd67"} Oct 31 00:57:48.951779 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:57:48.951727 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zpqhs/must-gather-62vpf" podStartSLOduration=1.5585846829999999 podStartE2EDuration="7.951708505s" totalImagesPullingTime="6.393123822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:48.951019987 +0000 UTC m=+1652.312602628" watchObservedRunningTime="2026-10-31 00:57:48.951708505 +0000 UTC m=+1652.313291134" Oct 31 00:57:49.840739 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:57:49.840700 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:58:09.009458 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:09.009425 5821 generic.go:356] "Generic (PLEG): container finished" podID="226286a2-cfc1-46a8-8383-41b449260d51" containerID="f2933c508f74a422aaf1901764ecd8cffc85b6cb76b12d1dc668dc3d9423dd67" exitCode=0 Oct 31 00:58:09.010056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:09.009502 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpqhs/must-gather-62vpf" event={"ID":"226286a2-cfc1-46a8-8383-41b449260d51","Type":"ContainerDied","Data":"f2933c508f74a422aaf1901764ecd8cffc85b6cb76b12d1dc668dc3d9423dd67"} Oct 31 00:58:09.010056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:09.009976 5821 scope.go:122] "RemoveContainer" containerID="f2933c508f74a422aaf1901764ecd8cffc85b6cb76b12d1dc668dc3d9423dd67" Oct 31 00:58:09.128129 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:09.128099 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zpqhs_must-gather-62vpf_226286a2-cfc1-46a8-8383-41b449260d51/gather/0.log" Oct 31 00:58:13.474557 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:13.474522 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wj84l_31d4d8b4-ee1c-4703-8ac5-f739d72ab13d/global-pull-secret-syncer/0.log" Oct 31 00:58:13.696369 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:13.696343 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d4jrp_089a2aaa-c90a-4df7-8e97-09650f16aab4/konnectivity-agent/0.log" Oct 31 00:58:13.806832 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:13.806765 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-252.us-east-2.compute.internal_703d740fe3de84816febee5a3376c36a/haproxy/0.log" Oct 31 00:58:14.831814 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:14.831776 5821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-zpqhs/must-gather-62vpf"] Oct 31 00:58:14.832304 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:14.832016 5821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-must-gather-zpqhs/must-gather-62vpf" podUID="226286a2-cfc1-46a8-8383-41b449260d51" containerName="copy" containerID="cri-o://e75d383acd848a479ddba8b9994c3eb6930be3d3f231ae6868ac5a47d81c380f" gracePeriod=2 Oct 31 00:58:14.836172 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:14.835766 5821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-zpqhs/must-gather-62vpf"] Oct 31 00:58:14.837228 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:58:14.837199 5821 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-62vpf\" is forbidden: User \"system:node:ip-10-0-134-252.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zpqhs\": no relationship found between node 'ip-10-0-134-252.us-east-2.compute.internal' and this object" podUID="226286a2-cfc1-46a8-8383-41b449260d51" pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:58:15.031125 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:15.031101 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zpqhs_must-gather-62vpf_226286a2-cfc1-46a8-8383-41b449260d51/copy/0.log" Oct 31 00:58:15.031497 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:15.031469 5821 generic.go:356] "Generic (PLEG): container finished" podID="226286a2-cfc1-46a8-8383-41b449260d51" containerID="e75d383acd848a479ddba8b9994c3eb6930be3d3f231ae6868ac5a47d81c380f" exitCode=143 Oct 31 00:58:15.055896 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:15.055878 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zpqhs_must-gather-62vpf_226286a2-cfc1-46a8-8383-41b449260d51/copy/0.log" Oct 31 00:58:15.058096 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:15.058079 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:58:15.060481 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:58:15.060460 5821 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-62vpf\" is forbidden: User \"system:node:ip-10-0-134-252.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zpqhs\": no relationship found between node 'ip-10-0-134-252.us-east-2.compute.internal' and this object" podUID="226286a2-cfc1-46a8-8383-41b449260d51" pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:58:15.125332 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:15.125272 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226286a2-cfc1-46a8-8383-41b449260d51-kube-api-access-55gmm\" (UniqueName: \"kubernetes.io/projected/226286a2-cfc1-46a8-8383-41b449260d51-kube-api-access-55gmm\") pod \"226286a2-cfc1-46a8-8383-41b449260d51\" (UID: \"226286a2-cfc1-46a8-8383-41b449260d51\") " Oct 31 00:58:15.125332 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:15.125311 5821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226286a2-cfc1-46a8-8383-41b449260d51-must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/226286a2-cfc1-46a8-8383-41b449260d51-must-gather-output\") pod \"226286a2-cfc1-46a8-8383-41b449260d51\" (UID: \"226286a2-cfc1-46a8-8383-41b449260d51\") " Oct 31 00:58:15.126735 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:15.126710 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226286a2-cfc1-46a8-8383-41b449260d51-must-gather-output" pod "226286a2-cfc1-46a8-8383-41b449260d51" (UID: "226286a2-cfc1-46a8-8383-41b449260d51"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:15.127479 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:15.127462 5821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226286a2-cfc1-46a8-8383-41b449260d51-kube-api-access-55gmm" pod "226286a2-cfc1-46a8-8383-41b449260d51" (UID: "226286a2-cfc1-46a8-8383-41b449260d51"). InnerVolumeSpecName "kube-api-access-55gmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:15.226499 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:15.226465 5821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55gmm\" (UniqueName: \"kubernetes.io/projected/226286a2-cfc1-46a8-8383-41b449260d51-kube-api-access-55gmm\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:15.226499 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:15.226492 5821 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/226286a2-cfc1-46a8-8383-41b449260d51-must-gather-output\") on node \"ip-10-0-134-252.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:16.036446 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:16.036413 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zpqhs_must-gather-62vpf_226286a2-cfc1-46a8-8383-41b449260d51/copy/0.log" Oct 31 00:58:16.036880 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:16.036826 5821 scope.go:122] "RemoveContainer" containerID="e75d383acd848a479ddba8b9994c3eb6930be3d3f231ae6868ac5a47d81c380f" Oct 31 00:58:16.036990 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:16.036829 5821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:58:16.039236 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:58:16.039206 5821 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-62vpf\" is forbidden: User \"system:node:ip-10-0-134-252.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zpqhs\": no relationship found between node 'ip-10-0-134-252.us-east-2.compute.internal' and this object" podUID="226286a2-cfc1-46a8-8383-41b449260d51" pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:58:16.044271 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:16.044253 5821 scope.go:122] "RemoveContainer" containerID="f2933c508f74a422aaf1901764ecd8cffc85b6cb76b12d1dc668dc3d9423dd67" Oct 31 00:58:16.046609 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:58:16.046583 5821 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-62vpf\" is forbidden: User \"system:node:ip-10-0-134-252.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zpqhs\": no relationship found between node 'ip-10-0-134-252.us-east-2.compute.internal' and this object" podUID="226286a2-cfc1-46a8-8383-41b449260d51" pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:58:16.777266 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:16.777227 5821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226286a2-cfc1-46a8-8383-41b449260d51" path="/var/lib/kubelet/pods/226286a2-cfc1-46a8-8383-41b449260d51/volumes" Oct 31 00:58:16.777497 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:58:16.777467 5821 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-62vpf\" is forbidden: User \"system:node:ip-10-0-134-252.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zpqhs\": no relationship found between node 'ip-10-0-134-252.us-east-2.compute.internal' and this object" podUID="226286a2-cfc1-46a8-8383-41b449260d51" pod="openshift-must-gather-zpqhs/must-gather-62vpf" Oct 31 00:58:19.339185 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:19.339151 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rcwt4_fba123f2-dd49-407a-8e1d-c568c0d66ce4/kube-state-metrics/0.log" Oct 31 00:58:19.371439 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:19.371416 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rcwt4_fba123f2-dd49-407a-8e1d-c568c0d66ce4/kube-rbac-proxy-main/0.log" Oct 31 00:58:19.404935 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:19.404902 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rcwt4_fba123f2-dd49-407a-8e1d-c568c0d66ce4/kube-rbac-proxy-self/0.log" Oct 31 00:58:19.483319 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:19.483294 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-rz5vm_9cd77f3f-3824-44dd-8d5f-156ff99d5b7e/monitoring-plugin/0.log" Oct 31 00:58:19.531146 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:19.531118 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7jm8z_b22fa0b3-c20e-402b-81b5-898a838d42c9/node-exporter/0.log" Oct 31 00:58:19.562731 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:19.562707 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7jm8z_b22fa0b3-c20e-402b-81b5-898a838d42c9/kube-rbac-proxy/0.log" Oct 31 00:58:19.604178 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:19.604115 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7jm8z_b22fa0b3-c20e-402b-81b5-898a838d42c9/init-textfile/0.log" Oct 31 00:58:19.838664 ip-10-0-134-252 kubenswrapper[5821]: E1031 00:58:19.838609 5821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g62jt" containerName="extractor" Oct 31 00:58:19.923974 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:19.923948 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-nv2qb_ea2acbfb-3c5b-4d40-8365-066e0d29f72e/kube-rbac-proxy-main/0.log" Oct 31 00:58:19.961472 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:19.961437 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-nv2qb_ea2acbfb-3c5b-4d40-8365-066e0d29f72e/kube-rbac-proxy-self/0.log" Oct 31 00:58:19.995877 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:19.995854 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-nv2qb_ea2acbfb-3c5b-4d40-8365-066e0d29f72e/openshift-state-metrics/0.log" Oct 31 00:58:20.046329 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:20.046298 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2a548c0-6c71-4ffa-9172-67bd7536a791/prometheus/0.log" Oct 31 00:58:20.077487 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:20.077466 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2a548c0-6c71-4ffa-9172-67bd7536a791/config-reloader/0.log" Oct 31 00:58:20.112397 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:20.112371 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2a548c0-6c71-4ffa-9172-67bd7536a791/thanos-sidecar/0.log" Oct 31 00:58:20.146221 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:20.146197 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2a548c0-6c71-4ffa-9172-67bd7536a791/kube-rbac-proxy-web/0.log" Oct 31 00:58:20.180348 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:20.180297 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2a548c0-6c71-4ffa-9172-67bd7536a791/kube-rbac-proxy/0.log" Oct 31 00:58:20.215366 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:20.215348 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2a548c0-6c71-4ffa-9172-67bd7536a791/kube-rbac-proxy-thanos/0.log" Oct 31 00:58:20.255535 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:20.255516 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2a548c0-6c71-4ffa-9172-67bd7536a791/init-config-reloader/0.log" Oct 31 00:58:21.320696 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.320661 5821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4"] Oct 31 00:58:21.321776 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.321095 5821 cpu_manager.go:414] "containerMap: removing container" podUID="226286a2-cfc1-46a8-8383-41b449260d51" containerName="gather" Oct 31 00:58:21.321776 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.321111 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226286a2-cfc1-46a8-8383-41b449260d51" containerName="gather" Oct 31 00:58:21.321776 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.321152 5821 cpu_manager.go:414] "containerMap: removing container" podUID="226286a2-cfc1-46a8-8383-41b449260d51" containerName="copy" Oct 31 00:58:21.321776 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.321159 5821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226286a2-cfc1-46a8-8383-41b449260d51" containerName="copy" Oct 31 00:58:21.321776 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.321222 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="226286a2-cfc1-46a8-8383-41b449260d51" containerName="gather" Oct 31 00:58:21.321776 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.321237 5821 memory_manager.go:365] "RemoveStaleState removing state" podUID="226286a2-cfc1-46a8-8383-41b449260d51" containerName="copy" Oct 31 00:58:21.326212 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.326193 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.338602 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.338581 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4"] Oct 31 00:58:21.344111 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.344092 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gxz57\"/\"kube-root-ca.crt\"" Oct 31 00:58:21.344929 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.344917 5821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gxz57\"/\"openshift-service-ca.crt\"" Oct 31 00:58:21.344989 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.344926 5821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gxz57\"/\"default-dockercfg-plgmx\"" Oct 31 00:58:21.373464 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.373442 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c3053a1-b6b8-419f-b666-bbd06f4848d9-sys\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.373566 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.373486 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c3053a1-b6b8-419f-b666-bbd06f4848d9-proc\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.373566 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.373509 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvzgx\" (UniqueName: \"kubernetes.io/projected/0c3053a1-b6b8-419f-b666-bbd06f4848d9-kube-api-access-fvzgx\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.373662 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.373608 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c3053a1-b6b8-419f-b666-bbd06f4848d9-lib-modules\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.373705 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.373670 5821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c3053a1-b6b8-419f-b666-bbd06f4848d9-podres\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.474877 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.474850 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c3053a1-b6b8-419f-b666-bbd06f4848d9-podres\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.475056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.474894 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c3053a1-b6b8-419f-b666-bbd06f4848d9-sys\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.475056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.474921 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c3053a1-b6b8-419f-b666-bbd06f4848d9-proc\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.475056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.474947 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvzgx\" (UniqueName: \"kubernetes.io/projected/0c3053a1-b6b8-419f-b666-bbd06f4848d9-kube-api-access-fvzgx\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.475056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.474976 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c3053a1-b6b8-419f-b666-bbd06f4848d9-podres\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.475056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.474986 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c3053a1-b6b8-419f-b666-bbd06f4848d9-sys\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.475056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.474979 5821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c3053a1-b6b8-419f-b666-bbd06f4848d9-lib-modules\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.475056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.475009 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c3053a1-b6b8-419f-b666-bbd06f4848d9-proc\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.475056 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.475034 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c3053a1-b6b8-419f-b666-bbd06f4848d9-lib-modules\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.484897 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.484875 5821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvzgx\" (UniqueName: \"kubernetes.io/projected/0c3053a1-b6b8-419f-b666-bbd06f4848d9-kube-api-access-fvzgx\") pod \"perf-node-gather-daemonset-rl2j4\" (UID: \"0c3053a1-b6b8-419f-b666-bbd06f4848d9\") " pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.638050 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.637963 5821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:21.908758 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:21.908683 5821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4"] Oct 31 00:58:21.912317 ip-10-0-134-252 kubenswrapper[5821]: W1031 00:58:21.912281 5821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0c3053a1_b6b8_419f_b666_bbd06f4848d9.slice/crio-eafa82d8ebb2ef476471cc083d62d767fa58687070b2703f3048b2372b4f50bc WatchSource:0}: Error finding container eafa82d8ebb2ef476471cc083d62d767fa58687070b2703f3048b2372b4f50bc: Status 404 returned error can't find the container with id eafa82d8ebb2ef476471cc083d62d767fa58687070b2703f3048b2372b4f50bc Oct 31 00:58:22.058458 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:22.058427 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" event={"ID":"0c3053a1-b6b8-419f-b666-bbd06f4848d9","Type":"ContainerStarted","Data":"fe9decc8d0ad6da02f9c76bf9eba7086e0fe7c13debac1f4554a0bb832c10ecc"} Oct 31 00:58:22.058588 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:22.058472 5821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" event={"ID":"0c3053a1-b6b8-419f-b666-bbd06f4848d9","Type":"ContainerStarted","Data":"eafa82d8ebb2ef476471cc083d62d767fa58687070b2703f3048b2372b4f50bc"} Oct 31 00:58:22.058588 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:22.058560 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:22.077696 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:22.077648 5821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" podStartSLOduration=1.077632132 podStartE2EDuration="1.077632132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:22.077197478 +0000 UTC m=+1685.438780117" watchObservedRunningTime="2026-10-31 00:58:22.077632132 +0000 UTC m=+1685.439214764" Oct 31 00:58:24.688239 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:24.688215 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xbdxj_97194d65-9d7e-47b2-8c55-2283301e8a4d/dns/0.log" Oct 31 00:58:24.721811 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:24.721790 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xbdxj_97194d65-9d7e-47b2-8c55-2283301e8a4d/kube-rbac-proxy/0.log" Oct 31 00:58:24.830581 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:24.830554 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m8d27_0b02c695-144a-4481-bc14-1cd89baba762/dns-node-resolver/0.log" Oct 31 00:58:25.424942 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:25.424913 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmn4m_fb2daade-8f58-42e8-a79b-2a373ff186ea/node-ca/0.log" Oct 31 00:58:26.797448 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:26.797424 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8ssc4_226527e6-3023-4c5a-b600-fb6322b77877/serve-healthcheck-canary/0.log" Oct 31 00:58:27.526591 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:27.526564 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g62jt_f79572d1-5d21-4771-b613-85a421e550b3/kube-rbac-proxy/0.log" Oct 31 00:58:27.559547 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:27.559520 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g62jt_f79572d1-5d21-4771-b613-85a421e550b3/exporter/0.log" Oct 31 00:58:27.597604 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:27.597578 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g62jt_f79572d1-5d21-4771-b613-85a421e550b3/extractor/0.log" Oct 31 00:58:28.071485 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:28.071456 5821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gxz57/perf-node-gather-daemonset-rl2j4" Oct 31 00:58:30.631513 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:30.631482 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7f9f8cdc65-flp4s_7a57c127-9e8c-482a-9511-d3959122f5be/manager/0.log" Oct 31 00:58:30.672267 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:30.672235 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9tj6m_b6dcf8ed-2f27-4841-9d44-985a060a4710/s3-init/0.log" Oct 31 00:58:30.700006 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:30.699988 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9tj6m_b6dcf8ed-2f27-4841-9d44-985a060a4710/download-hf-model/0.log" Oct 31 00:58:30.740498 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:30.740474 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-78bb9c6ff8-j7brj_9547691d-0f50-476f-a4c1-8f084638d517/seaweedfs/0.log" Oct 31 00:58:37.029272 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:37.029240 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9zlgx_db73288c-815d-49ed-9cdb-6799fb6ecc7b/kube-multus/0.log" Oct 31 00:58:37.059516 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:37.059489 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqkmw_50ef0d54-5212-427c-8bab-04b4262c3e11/kube-multus-additional-cni-plugins/0.log" Oct 31 00:58:37.091810 ip-10-0-134-252 kubenswrapper[5821]: I1031 00:58:37.091785 5821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqkmw_50ef0d54-5212-427c-8bab-04b4262c3e11/egress-router-binary-copy/0.log"