Jul 23 01:28:07.313953 ip-10-0-131-117 systemd[1]: Starting Kubernetes Kubelet... Jul 23 01:28:07.769164 ip-10-0-131-117 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 01:28:07.769164 ip-10-0-131-117 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 01:28:07.769164 ip-10-0-131-117 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 01:28:07.769164 ip-10-0-131-117 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 01:28:07.769164 ip-10-0-131-117 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 01:28:07.772005 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.771911 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 01:28:07.777213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777195 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 01:28:07.777213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777212 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 01:28:07.777213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777216 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777220 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777223 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777226 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777229 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777239 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777242 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777245 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777247 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777250 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777253 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777255 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777258 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777262 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777266 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777268 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777271 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777274 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777276 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 01:28:07.777315 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777279 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777282 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777285 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777288 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777290 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777292 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777295 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777298 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777300 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777303 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777305 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777308 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777310 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777313 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777316 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777319 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777322 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777325 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777328 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777331 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 01:28:07.777817 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777334 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777337 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777340 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777342 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777346 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777350 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777355 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777358 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777361 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777364 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777366 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777369 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777372 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777375 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777378 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777381 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777383 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777386 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777389 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 01:28:07.778282 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777391 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777393 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777397 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777399 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777403 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777406 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777408 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777411 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777414 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777417 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777419 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777423 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777425 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777428 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777430 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777433 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777435 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777438 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777440 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777443 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 01:28:07.778746 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777446 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777448 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777451 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777453 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777456 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777459 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777462 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777465 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777468 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777470 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777473 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777475 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777478 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777480 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777482 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777485 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777488 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777490 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 01:28:07.779213 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.777493 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 01:28:07.779885 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779871 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 23 01:28:07.779885 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779885 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779891 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779894 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779897 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779901 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779903 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779906 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779909 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779912 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779914 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779917 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779919 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779922 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779925 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779927 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779930 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779933 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779936 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779938 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779941 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 01:28:07.779944 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779943 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779946 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779949 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779952 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779955 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779965 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779968 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779971 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779973 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779976 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779978 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779981 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779984 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779987 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779989 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779992 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779994 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.779998 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780001 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780003 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 01:28:07.780476 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780006 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780008 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780011 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780013 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780016 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780018 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780020 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780023 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780025 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780028 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780030 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780032 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780035 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780037 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780041 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780044 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780046 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780049 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780052 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780054 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 01:28:07.780976 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780056 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780059 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780061 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780064 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780066 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780069 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780071 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780074 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780076 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780078 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780084 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780087 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780090 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780092 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780095 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780098 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780100 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780102 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780105 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 01:28:07.781468 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780109 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780113 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780116 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780119 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780122 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780124 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780127 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780130 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780133 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780136 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780140 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780143 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780145 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780148 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780151 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780153 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780157 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780159 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780163 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780542 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 01:28:07.781947 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780547 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780551 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780554 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780559 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780563 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780567 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780570 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780573 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780576 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780579 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780582 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780584 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780587 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780591 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780594 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780597 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780599 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780602 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 01:28:07.782428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780605 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780608 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780610 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780613 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780616 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780618 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780621 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780623 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780626 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780628 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780645 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780649 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780652 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780654 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780657 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780660 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780662 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780665 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780669 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780671 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 01:28:07.782893 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780674 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780677 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780679 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780682 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780684 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780687 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780690 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780692 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780695 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780697 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780700 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780702 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780705 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780708 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780711 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780713 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780716 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780718 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780721 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780724 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 01:28:07.783356 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780726 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780728 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780734 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780736 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780739 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780741 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780744 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780746 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780749 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780751 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780754 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780757 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780760 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780762 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780765 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780768 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780770 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780774 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780776 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780779 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 01:28:07.783848 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780781 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780784 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780786 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780789 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780791 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780796 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780798 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780801 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780804 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780806 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780809 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780811 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780814 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780817 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780821 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780823 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780826 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780830 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780833 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.780836 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 01:28:07.784440 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780909 2572 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780916 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780923 2572 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780927 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780932 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780935 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780940 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780944 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780948 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780951 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780954 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780958 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780961 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780964 2572 flags.go:64] FLAG: --cgroup-root="" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780967 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780970 2572 flags.go:64] FLAG: --client-ca-file="" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780973 2572 flags.go:64] FLAG: --cloud-provider="external" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780978 2572 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780983 2572 flags.go:64] FLAG: --cluster-domain="" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780986 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780990 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780993 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.780996 2572 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781000 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 01:28:07.784947 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781003 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781006 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781010 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781013 2572 flags.go:64] FLAG: --contention-profiling="false" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781016 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781020 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781023 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781026 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781030 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781033 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781036 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781039 2572 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781042 2572 flags.go:64] FLAG: --enable-server="true" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781048 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781052 2572 flags.go:64] FLAG: --event-burst="100" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781056 2572 flags.go:64] FLAG: --event-qps="50" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781059 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781062 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781064 2572 flags.go:64] FLAG: --eviction-hard="" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781068 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781071 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781074 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781077 2572 flags.go:64] FLAG: --eviction-soft="" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781080 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781083 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 01:28:07.785517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781086 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781089 2572 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781092 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781095 2572 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781098 2572 flags.go:64] FLAG: --feature-gates="" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781101 2572 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781105 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781123 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781127 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781131 2572 flags.go:64] FLAG: --healthz-port="10248" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781134 2572 flags.go:64] FLAG: --help="false" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781137 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-131-117.ec2.internal" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781141 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781144 2572 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781147 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781150 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781153 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781157 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781160 2572 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781162 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781165 2572 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781169 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781172 2572 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781175 2572 flags.go:64] FLAG: --kube-reserved="" Jul 23 01:28:07.786190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781178 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781181 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781184 2572 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781187 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781189 2572 flags.go:64] FLAG: --lock-file="" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781192 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781195 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781198 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781204 2572 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781206 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781209 2572 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781212 2572 flags.go:64] FLAG: --logging-format="text" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781215 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781218 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781221 2572 flags.go:64] FLAG: --manifest-url="" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781224 2572 flags.go:64] FLAG: --manifest-url-header="" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781228 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781231 2572 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781235 2572 flags.go:64] FLAG: --max-pods="110" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781238 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781241 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781244 2572 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781247 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781250 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781253 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 01:28:07.786766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781256 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781264 2572 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781267 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781270 2572 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781273 2572 flags.go:64] FLAG: --pod-cidr="" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781276 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781282 2572 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781285 2572 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781288 2572 flags.go:64] FLAG: --pods-per-core="0" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781291 2572 flags.go:64] FLAG: --port="10250" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781294 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781297 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06d0320a35f7c4e28" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781300 2572 flags.go:64] FLAG: --qos-reserved="" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781303 2572 flags.go:64] FLAG: --read-only-port="10255" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781306 2572 flags.go:64] FLAG: --register-node="true" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781309 2572 flags.go:64] FLAG: --register-with-taints="" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781313 2572 flags.go:64] FLAG: --registry-burst="10" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781315 2572 flags.go:64] FLAG: --registry-qps="5" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781318 2572 flags.go:64] FLAG: --reserved-cpus="" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781322 2572 flags.go:64] FLAG: --reserved-memory="" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781325 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781329 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781332 2572 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781335 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781338 2572 flags.go:64] FLAG: --runonce="false" Jul 23 01:28:07.787368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781341 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781344 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781347 2572 flags.go:64] FLAG: --seccomp-default="false" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781349 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781352 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781355 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781358 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781361 2572 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781364 2572 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781367 2572 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781370 2572 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781373 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781376 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781379 2572 flags.go:64] FLAG: --system-cgroups="" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781382 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781387 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781390 2572 flags.go:64] FLAG: --tls-cert-file="" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781393 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781398 2572 flags.go:64] FLAG: --tls-min-version="" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781401 2572 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781403 2572 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781407 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781410 2572 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781413 2572 flags.go:64] FLAG: --v="2" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781416 2572 flags.go:64] FLAG: --version="false" Jul 23 01:28:07.787994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781420 2572 flags.go:64] FLAG: --vmodule="" Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781424 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.781427 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781542 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781547 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781551 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781555 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781558 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781561 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781564 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781567 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781571 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781575 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781578 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781581 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781585 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781588 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781591 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781594 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 01:28:07.788623 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781597 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781599 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781602 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781605 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781608 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781610 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781613 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781616 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781618 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781621 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781624 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781626 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781629 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781643 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781646 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781649 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781652 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781654 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781657 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781661 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 01:28:07.789096 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781664 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781667 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781669 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781672 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781675 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781677 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781680 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781683 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781687 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781690 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781693 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781695 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781698 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781700 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781703 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781707 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781710 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781712 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781715 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781718 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 01:28:07.789619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781720 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781723 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781726 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781728 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781731 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781733 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781736 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781738 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781741 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781744 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781747 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781751 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781754 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781758 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781760 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781763 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781766 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781769 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781772 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 01:28:07.790115 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781774 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781779 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781781 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781784 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781787 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781789 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781792 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781795 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781797 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781800 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781803 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781805 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781808 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781811 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781813 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781816 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781819 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781822 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781826 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781829 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 01:28:07.790567 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781831 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781834 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781837 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.781839 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.782581 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.789791 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.789807 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.789827 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.789836 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789899 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789905 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789909 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789912 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789915 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 01:28:07.791357 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789918 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789921 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789924 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789927 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789930 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789933 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789936 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789938 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789941 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789943 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789946 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789948 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789951 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789953 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789956 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789959 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789961 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789964 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789966 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789969 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 01:28:07.792027 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789971 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789974 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789976 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789979 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789983 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789986 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789990 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789993 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789997 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.789999 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790002 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790004 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790007 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790009 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790012 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790015 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790017 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790020 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790022 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 01:28:07.792524 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790024 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790027 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790030 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790032 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790035 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790039 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790042 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790046 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790049 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790052 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790055 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790058 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790061 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790063 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790066 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790068 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790071 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790074 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790079 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790083 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 01:28:07.793036 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790085 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790088 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790091 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790094 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790096 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790098 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790101 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790104 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790106 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790108 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790111 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790114 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790116 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790119 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790122 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790124 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790127 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790130 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790132 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790135 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 01:28:07.793519 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790137 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790140 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790142 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790145 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790147 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790150 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790152 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790155 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790158 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790161 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790164 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790167 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790170 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790173 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 01:28:07.794029 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790175 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.790180 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790294 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790299 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790302 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790304 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790309 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790311 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790314 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790317 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790319 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790322 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790325 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790327 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 01:28:07.794365 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790330 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790333 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790336 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790338 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790341 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790343 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790346 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790349 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790351 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790354 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790357 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790359 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790362 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790365 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790368 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790371 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790374 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790376 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790379 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790381 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 01:28:07.794728 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790383 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790386 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790388 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790391 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790393 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790396 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790398 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790401 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790403 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790406 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790410 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790413 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790416 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790419 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790422 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790424 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790427 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790429 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790432 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 01:28:07.795215 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790435 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790437 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790440 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790442 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790445 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790447 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790450 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790453 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790456 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790458 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790461 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790464 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790466 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790469 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790472 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790474 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790477 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790480 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790482 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790485 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 01:28:07.795694 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790487 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790489 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790492 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790494 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790497 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790499 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790502 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790504 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790507 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790509 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790511 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790514 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790516 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790518 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790521 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790524 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790526 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790528 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790531 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790535 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 01:28:07.796188 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790538 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790541 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790544 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790547 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790551 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790554 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790558 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:07.790561 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.790566 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.791390 2572 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.793865 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.795695 2572 server.go:1019] "Starting client certificate rotation" Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.795797 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 01:28:07.796680 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.795827 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 01:28:07.799277 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.799264 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 01:28:07.800439 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.800421 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 01:28:07.809095 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.809075 2572 log.go:25] "Validated CRI v1 runtime API" Jul 23 01:28:07.811593 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.811578 2572 log.go:25] "Validated CRI v1 image API" Jul 23 01:28:07.812375 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.812361 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 01:28:07.815044 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.815026 2572 fs.go:135] Filesystem UUIDs: map[35664ddb-16e7-43f3-b261-1b659812cdb2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9328e2ba-ea93-4aab-b840-168d3fe3d580:/dev/nvme0n1p3] Jul 23 01:28:07.815112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.815043 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 01:28:07.821067 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.820958 2572 manager.go:216] Machine: {Timestamp:2026-07-23 01:28:07.81961508 +0000 UTC m=+0.364564244 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101239 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a174675e3e13f8dbc813954b564bc SystemUUID:ec2a1746-75e3-e13f-8dbc-813954b564bc BootID:d1cd8843-100f-486d-9d21-f9cbd54d78cc Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:62:a5:ab:c0:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:62:a5:ab:c0:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:b9:a1:98:2b:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 01:28:07.821067 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.821061 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 01:28:07.821188 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.821137 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 01:28:07.824729 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.824710 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 01:28:07.824806 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.824780 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 01:28:07.824944 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.824812 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-117.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 01:28:07.826228 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.826216 2572 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 01:28:07.826284 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.826231 2572 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 01:28:07.826284 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.826238 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 01:28:07.827030 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.827020 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 01:28:07.828837 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.828827 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 01:28:07.832716 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.832705 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 23 01:28:07.832831 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.832821 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 01:28:07.832868 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.832855 2572 kubelet.go:486] "Attempting to sync node with API server" Jul 23 01:28:07.832899 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.832871 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 01:28:07.832899 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.832889 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 01:28:07.832952 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.832900 2572 kubelet.go:398] "Adding apiserver pod source" Jul 23 01:28:07.832952 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.832909 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 01:28:07.833309 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.833262 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 01:28:07.833309 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.833279 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 01:28:07.837718 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.837696 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 01:28:07.840317 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.840302 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 01:28:07.840379 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.840333 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 01:28:07.841777 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.841765 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 01:28:07.841815 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.841782 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 01:28:07.841815 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.841789 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 01:28:07.841815 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.841795 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 01:28:07.841815 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.841804 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 01:28:07.841815 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.841809 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 01:28:07.841815 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.841817 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 01:28:07.842002 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.841823 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 01:28:07.842002 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.841829 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 01:28:07.842002 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.841834 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 01:28:07.842002 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.841843 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 01:28:07.842002 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.841848 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 01:28:07.842145 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:07.842101 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 01:28:07.842238 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:07.842217 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 01:28:07.843627 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.843612 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 01:28:07.843627 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.843624 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 01:28:07.845214 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.845196 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-117.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 01:28:07.846709 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.846696 2572 server.go:1268] "Started kubelet" Jul 23 01:28:07.846821 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.846769 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 01:28:07.846888 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.846805 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 01:28:07.846888 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.846876 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 01:28:07.847563 ip-10-0-131-117 systemd[1]: Started Kubernetes Kubelet. Jul 23 01:28:07.848081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.848054 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5ptdd" Jul 23 01:28:07.849057 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.849040 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 01:28:07.850554 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.850463 2572 server.go:310] "Adding debug handlers to kubelet server" Jul 23 01:28:07.851315 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:07.851298 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 01:28:07.856551 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.856533 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5ptdd" Jul 23 01:28:07.858247 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.858225 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 01:28:07.858330 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.858225 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 01:28:07.858716 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:07.858446 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:07.859000 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.858982 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 01:28:07.859090 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.859014 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 01:28:07.859090 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.859030 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 01:28:07.859090 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.859063 2572 factory.go:55] Registering systemd factory Jul 23 01:28:07.859090 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.859078 2572 factory.go:223] Registration of the systemd container factory successfully Jul 23 01:28:07.859301 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.859161 2572 reconstruct.go:97] "Volume reconstruction finished" Jul 23 01:28:07.859301 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.859168 2572 reconciler.go:29] "Reconciler: start to sync state" Jul 23 01:28:07.859301 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:07.858149 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-117.ec2.internal.18c4c6f053d7ef31 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-117.ec2.internal,UID:ip-10-0-131-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-117.ec2.internal,},FirstTimestamp:2026-07-23 01:28:07.846670129 +0000 UTC m=+0.391619293,LastTimestamp:2026-07-23 01:28:07.846670129 +0000 UTC m=+0.391619293,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-117.ec2.internal,}" Jul 23 01:28:07.859414 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.859363 2572 factory.go:153] Registering CRI-O factory Jul 23 01:28:07.859414 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.859377 2572 factory.go:223] Registration of the crio container factory successfully Jul 23 01:28:07.859475 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.859427 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 01:28:07.859475 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.859456 2572 factory.go:103] Registering Raw factory Jul 23 01:28:07.859475 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.859469 2572 manager.go:1196] Started watching for new ooms in manager Jul 23 01:28:07.859899 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.859887 2572 manager.go:349] Starting recovery of all containers Jul 23 01:28:07.867068 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.866385 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 01:28:07.869950 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:07.869921 2572 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-117.ec2.internal\" not found" node="ip-10-0-131-117.ec2.internal" Jul 23 01:28:07.870737 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.870718 2572 manager.go:354] Recovery completed Jul 23 01:28:07.876705 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.876692 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:07.878786 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.878768 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:07.878874 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.878801 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:07.878874 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.878815 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:07.879367 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.879352 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 01:28:07.879367 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.879363 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 01:28:07.879486 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.879382 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 23 01:28:07.879486 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.879396 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:07.880342 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.880323 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:07.880407 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.880357 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:07.880407 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.880371 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:07.880604 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.880593 2572 policy_none.go:49] "None policy: Start" Jul 23 01:28:07.880650 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.880610 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 01:28:07.880650 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.880619 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 01:28:07.881467 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.881457 2572 policy_none.go:47] "Start" Jul 23 01:28:07.921652 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.921390 2572 manager.go:337] "Starting Device Plugin manager" Jul 23 01:28:07.932756 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:07.921671 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 01:28:07.932756 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.921685 2572 server.go:87] "Starting device plugin registration server" Jul 23 01:28:07.932756 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.921997 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 01:28:07.932756 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.922009 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 01:28:07.932756 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.922098 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 01:28:07.932756 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.922194 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 01:28:07.932756 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.922202 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 01:28:07.932756 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:07.922901 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 01:28:07.932756 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:07.922958 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:07.967616 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.967571 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 01:28:07.968963 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.968944 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 01:28:07.969052 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.968994 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 01:28:07.969052 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.969020 2572 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 01:28:07.969143 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:07.969062 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 01:28:07.970895 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:07.970879 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 01:28:08.022793 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.022701 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:08.023666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.023627 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:08.023771 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.023679 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:08.023771 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.023693 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:08.023771 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.023724 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.036350 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.036330 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.036430 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.036358 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-117.ec2.internal\": node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:08.053338 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.053311 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:08.069958 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.069935 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-117.ec2.internal"] Jul 23 01:28:08.070093 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.069996 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:08.070918 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.070902 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:08.071000 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.070931 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:08.071000 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.070946 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:08.072304 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.072291 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:08.072427 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.072412 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.072497 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.072442 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:08.073018 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.072999 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:08.073112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.073027 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:08.073112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.073037 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:08.073112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.073008 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:08.073112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.073094 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:08.073112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.073107 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:08.074318 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.074302 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.074395 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.074326 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 01:28:08.074951 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.074932 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientMemory" Jul 23 01:28:08.075040 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.074967 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 01:28:08.075040 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.074985 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeHasSufficientPID" Jul 23 01:28:08.099460 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.099436 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-117.ec2.internal\" not found" node="ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.102866 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.102852 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-117.ec2.internal\" not found" node="ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.153649 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.153618 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:08.254099 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.254067 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:08.260332 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.260311 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d368fcab28cf50f84a474490a3004fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal\" (UID: \"d368fcab28cf50f84a474490a3004fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.260390 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.260337 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d368fcab28cf50f84a474490a3004fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal\" (UID: \"d368fcab28cf50f84a474490a3004fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.260390 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.260357 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09f69a889a832c91f7f75eb1f3452d24-config\") pod \"kube-apiserver-proxy-ip-10-0-131-117.ec2.internal\" (UID: \"09f69a889a832c91f7f75eb1f3452d24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.354744 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.354670 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:08.361006 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.360981 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d368fcab28cf50f84a474490a3004fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal\" (UID: \"d368fcab28cf50f84a474490a3004fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.361059 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.361014 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09f69a889a832c91f7f75eb1f3452d24-config\") pod \"kube-apiserver-proxy-ip-10-0-131-117.ec2.internal\" (UID: \"09f69a889a832c91f7f75eb1f3452d24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.361102 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.361030 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d368fcab28cf50f84a474490a3004fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal\" (UID: \"d368fcab28cf50f84a474490a3004fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.361102 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.361072 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d368fcab28cf50f84a474490a3004fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal\" (UID: \"d368fcab28cf50f84a474490a3004fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.361102 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.361090 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09f69a889a832c91f7f75eb1f3452d24-config\") pod \"kube-apiserver-proxy-ip-10-0-131-117.ec2.internal\" (UID: \"09f69a889a832c91f7f75eb1f3452d24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.361203 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.361102 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d368fcab28cf50f84a474490a3004fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal\" (UID: \"d368fcab28cf50f84a474490a3004fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.403153 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.403120 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.405249 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.405223 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.ec2.internal" Jul 23 01:28:08.455457 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.455425 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:08.555907 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.555880 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:08.656694 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.656613 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:08.757773 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.757740 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:08.795057 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.795016 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 01:28:08.795513 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.795207 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 01:28:08.795513 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.795222 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 01:28:08.857853 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.857819 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:08.862292 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.862250 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 01:23:08 +0000 UTC" deadline="2028-01-21 00:47:16.21986443 +0000 UTC" Jul 23 01:28:08.862292 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.862292 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13127h19m7.357577709s" Jul 23 01:28:08.919937 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:08.919895 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f69a889a832c91f7f75eb1f3452d24.slice/crio-d75987ae5d83ee247f774e6df4602e1478bcf65890c6c1ce86a5a5cc821c2c09 WatchSource:0}: Error finding container d75987ae5d83ee247f774e6df4602e1478bcf65890c6c1ce86a5a5cc821c2c09: Status 404 returned error can't find the container with id d75987ae5d83ee247f774e6df4602e1478bcf65890c6c1ce86a5a5cc821c2c09 Jul 23 01:28:08.920425 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:08.920393 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd368fcab28cf50f84a474490a3004fef.slice/crio-63c5d09583093ee8172d98c90f7b6121e522c1ccafe0f3a76e7771b5f065c43e WatchSource:0}: Error finding container 63c5d09583093ee8172d98c90f7b6121e522c1ccafe0f3a76e7771b5f065c43e: Status 404 returned error can't find the container with id 63c5d09583093ee8172d98c90f7b6121e522c1ccafe0f3a76e7771b5f065c43e Jul 23 01:28:08.925904 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.925885 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 01:28:08.958837 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:08.958799 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:08.975913 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.975864 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.ec2.internal" event={"ID":"09f69a889a832c91f7f75eb1f3452d24","Type":"ContainerStarted","Data":"d75987ae5d83ee247f774e6df4602e1478bcf65890c6c1ce86a5a5cc821c2c09"} Jul 23 01:28:08.976991 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:08.976970 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" event={"ID":"d368fcab28cf50f84a474490a3004fef","Type":"ContainerStarted","Data":"63c5d09583093ee8172d98c90f7b6121e522c1ccafe0f3a76e7771b5f065c43e"} Jul 23 01:28:09.059959 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:09.059921 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-117.ec2.internal\" not found" Jul 23 01:28:09.086625 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.086602 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 01:28:09.112235 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.112212 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 01:28:09.160559 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.160461 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" Jul 23 01:28:09.173533 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.173502 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 01:28:09.174513 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.174498 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.ec2.internal" Jul 23 01:28:09.180828 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.180811 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 01:28:09.616962 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.616881 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 01:28:09.834510 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.834444 2572 apiserver.go:52] "Watching apiserver" Jul 23 01:28:09.846420 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.846383 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 01:28:09.846964 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.846933 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-j5hcq","openshift-multus/network-metrics-daemon-sgszg","openshift-network-operator/iptables-alerter-hnqjc","kube-system/konnectivity-agent-zvjp5","kube-system/kube-apiserver-proxy-ip-10-0-131-117.ec2.internal","openshift-image-registry/node-ca-rz68t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal","openshift-network-diagnostics/network-check-target-dl8vz","openshift-ovn-kubernetes/ovnkube-node-f6v4s","kube-system/global-pull-secret-syncer-vvpsg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg","openshift-cluster-node-tuning-operator/tuned-srw7p","openshift-dns/node-resolver-ssx5d","openshift-multus/multus-additional-cni-plugins-xfprg"] Jul 23 01:28:09.849034 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.849007 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:09.849150 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:09.849098 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:09.850428 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.850407 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.852042 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.852016 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hnqjc" Jul 23 01:28:09.853066 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.853042 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pfn6r\"" Jul 23 01:28:09.853426 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.853408 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 01:28:09.853626 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.853608 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 01:28:09.853976 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.853952 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 01:28:09.854248 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.854231 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zvjp5" Jul 23 01:28:09.854386 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.854363 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 01:28:09.854771 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.854748 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 01:28:09.854858 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.854785 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6gjdg\"" Jul 23 01:28:09.854858 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.854789 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 01:28:09.856666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.856626 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.857643 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.857610 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sfsxb\"" Jul 23 01:28:09.857935 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.857820 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 01:28:09.857935 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.857898 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 01:28:09.860772 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.858433 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rz68t" Jul 23 01:28:09.860772 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.859209 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 01:28:09.860772 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.859223 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 01:28:09.860772 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.859494 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 01:28:09.860772 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.859762 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 01:28:09.860772 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.859981 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2btv7\"" Jul 23 01:28:09.860772 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.860047 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 01:28:09.860772 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.860582 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.861933 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.861893 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 01:28:09.861933 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.861932 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 01:28:09.862103 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.862077 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 01:28:09.862170 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.862139 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 01:28:09.862224 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.862170 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zskbn\"" Jul 23 01:28:09.863071 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.862539 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 01:28:09.863071 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.862595 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ssx5d" Jul 23 01:28:09.863071 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.862810 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 01:28:09.864401 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.864381 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:09.864507 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:09.864465 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:09.865415 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.865265 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 01:28:09.865526 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.865504 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 01:28:09.866132 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.865682 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dhn5g\"" Jul 23 01:28:09.866132 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.865695 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ch6k7\"" Jul 23 01:28:09.866132 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.865777 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 01:28:09.866132 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.865696 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 01:28:09.866132 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.865943 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 01:28:09.866814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.866597 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 01:28:09.868048 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.867337 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.869675 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.869577 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:09.869675 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:09.869638 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.870582 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-46kww\"" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.870784 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871046 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-device-dir\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53c3ed25-3b0b-46b0-a5ba-1b0b87900743-konnectivity-ca\") pod \"konnectivity-agent-zvjp5\" (UID: \"53c3ed25-3b0b-46b0-a5ba-1b0b87900743\") " pod="kube-system/konnectivity-agent-zvjp5" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871104 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50570df0-9213-4039-ab5c-a1347dc88854-host\") pod \"node-ca-rz68t\" (UID: \"50570df0-9213-4039-ab5c-a1347dc88854\") " pod="openshift-image-registry/node-ca-rz68t" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871131 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-run-netns\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871147 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871157 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-run-ovn\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871180 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f94eece0-4626-4e4e-8c9f-d8aab764fe95-dbus\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871204 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edab0343-4377-4ac0-8106-b39733beefc7-cnibin\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871245 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-cni-bin\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871271 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-sys-fs\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871295 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-slash\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871320 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d5c6a0a-a492-4aa6-b28b-ce5da66c8079-host-slash\") pod \"iptables-alerter-hnqjc\" (UID: \"5d5c6a0a-a492-4aa6-b28b-ce5da66c8079\") " pod="openshift-network-operator/iptables-alerter-hnqjc" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871344 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edab0343-4377-4ac0-8106-b39733beefc7-system-cni-dir\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edab0343-4377-4ac0-8106-b39733beefc7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871397 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.872179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871434 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-kubelet\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871460 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-etc-openvswitch\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871483 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-run-openvswitch\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871517 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871555 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-cni-netd\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871582 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fa70596-1837-4f06-b94c-88558d60b65e-ovnkube-script-lib\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871613 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d5c6a0a-a492-4aa6-b28b-ce5da66c8079-iptables-alerter-script\") pod \"iptables-alerter-hnqjc\" (UID: \"5d5c6a0a-a492-4aa6-b28b-ce5da66c8079\") " pod="openshift-network-operator/iptables-alerter-hnqjc" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871654 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlk99\" (UniqueName: \"kubernetes.io/projected/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-kube-api-access-xlk99\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871685 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-systemd-units\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871740 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-var-lib-openvswitch\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871771 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871799 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7pq4\" (UniqueName: \"kubernetes.io/projected/5d5c6a0a-a492-4aa6-b28b-ce5da66c8079-kube-api-access-p7pq4\") pod \"iptables-alerter-hnqjc\" (UID: \"5d5c6a0a-a492-4aa6-b28b-ce5da66c8079\") " pod="openshift-network-operator/iptables-alerter-hnqjc" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871826 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/edab0343-4377-4ac0-8106-b39733beefc7-os-release\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871854 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hglj8\" (UniqueName: \"kubernetes.io/projected/edab0343-4377-4ac0-8106-b39733beefc7-kube-api-access-hglj8\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871886 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-node-log\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871915 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fa70596-1837-4f06-b94c-88558d60b65e-ovn-node-metrics-cert\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.873267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871956 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edab0343-4377-4ac0-8106-b39733beefc7-cni-binary-copy\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.871992 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-registration-dir\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872022 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53c3ed25-3b0b-46b0-a5ba-1b0b87900743-agent-certs\") pod \"konnectivity-agent-zvjp5\" (UID: \"53c3ed25-3b0b-46b0-a5ba-1b0b87900743\") " pod="kube-system/konnectivity-agent-zvjp5" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/50570df0-9213-4039-ab5c-a1347dc88854-serviceca\") pod \"node-ca-rz68t\" (UID: \"50570df0-9213-4039-ab5c-a1347dc88854\") " pod="openshift-image-registry/node-ca-rz68t" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872089 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99mm6\" (UniqueName: \"kubernetes.io/projected/50570df0-9213-4039-ab5c-a1347dc88854-kube-api-access-99mm6\") pod \"node-ca-rz68t\" (UID: \"50570df0-9213-4039-ab5c-a1347dc88854\") " pod="openshift-image-registry/node-ca-rz68t" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872151 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-log-socket\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fa70596-1837-4f06-b94c-88558d60b65e-env-overrides\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872207 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l42b\" (UniqueName: \"kubernetes.io/projected/2fa70596-1837-4f06-b94c-88558d60b65e-kube-api-access-9l42b\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872253 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f94eece0-4626-4e4e-8c9f-d8aab764fe95-kubelet-config\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872277 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872305 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872331 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fa70596-1837-4f06-b94c-88558d60b65e-ovnkube-config\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872366 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edab0343-4377-4ac0-8106-b39733beefc7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872394 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-run-systemd\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872419 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edab0343-4377-4ac0-8106-b39733beefc7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.872445 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-socket-dir\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.873824 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 01:28:09.874122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.874050 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bbv27\"" Jul 23 01:28:09.879337 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.879109 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 01:28:09.893203 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.893134 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 01:28:09.960173 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.960141 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 01:28:09.972652 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.972602 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-systemd\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.972841 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.972692 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-lib-modules\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.972841 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.972732 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-var-lib-kubelet\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.972841 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.972775 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-system-cni-dir\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.972841 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.972822 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a59326e-1357-42d7-96ae-4d6e43dabf24-multus-daemon-config\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.973050 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.972854 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-host\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.973050 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.972887 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-device-dir\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.973050 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.972914 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-sys-fs\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.973050 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.972969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edab0343-4377-4ac0-8106-b39733beefc7-system-cni-dir\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.973050 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.972998 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbj8q\" (UniqueName: \"kubernetes.io/projected/820cce87-a018-4751-a835-a045e7ccfd8e-kube-api-access-fbj8q\") pod \"node-resolver-ssx5d\" (UID: \"820cce87-a018-4751-a835-a045e7ccfd8e\") " pod="openshift-dns/node-resolver-ssx5d" Jul 23 01:28:09.973050 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.972996 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-device-dir\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.973050 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973023 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-sysconfig\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973055 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-sys-fs\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-kubelet\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-etc-openvswitch\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973128 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-run-openvswitch\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973134 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-etc-openvswitch\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973072 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edab0343-4377-4ac0-8106-b39733beefc7-system-cni-dir\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973133 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-kubelet\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973154 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973183 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-run-openvswitch\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973193 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-cni-netd\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973219 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fa70596-1837-4f06-b94c-88558d60b65e-ovnkube-script-lib\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglj8\" (UniqueName: \"kubernetes.io/projected/edab0343-4377-4ac0-8106-b39733beefc7-kube-api-access-hglj8\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-cni-netd\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973262 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973275 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-kubernetes\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.973406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973408 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlk99\" (UniqueName: \"kubernetes.io/projected/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-kube-api-access-xlk99\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973442 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973504 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7pq4\" (UniqueName: \"kubernetes.io/projected/5d5c6a0a-a492-4aa6-b28b-ce5da66c8079-kube-api-access-p7pq4\") pod \"iptables-alerter-hnqjc\" (UID: \"5d5c6a0a-a492-4aa6-b28b-ce5da66c8079\") " pod="openshift-network-operator/iptables-alerter-hnqjc" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973533 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-cnibin\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:09.973663 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973666 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-run-multus-certs\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973706 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-sysctl-d\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:09.973763 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret podName:f94eece0-4626-4e4e-8c9f-d8aab764fe95 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:10.473717792 +0000 UTC m=+3.018666966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret") pod "global-pull-secret-syncer-vvpsg" (UID: "f94eece0-4626-4e4e-8c9f-d8aab764fe95") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973788 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/842be287-d8ca-4bf4-9974-091b54c7191c-tmp\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973823 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973853 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-node-log\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fa70596-1837-4f06-b94c-88558d60b65e-ovn-node-metrics-cert\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973930 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-node-log\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973968 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-var-lib-cni-bin\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.973999 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h6wv\" (UniqueName: \"kubernetes.io/projected/2a59326e-1357-42d7-96ae-4d6e43dabf24-kube-api-access-5h6wv\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974021 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fa70596-1837-4f06-b94c-88558d60b65e-ovnkube-script-lib\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.974299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974030 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmhcr\" (UniqueName: \"kubernetes.io/projected/62b44b95-e27a-42a9-b058-0e0363718750-kube-api-access-kmhcr\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974057 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8kxq\" (UniqueName: \"kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq\") pod \"network-check-target-dl8vz\" (UID: \"a20bf778-ada3-487a-b3e2-946cbf2366e2\") " pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974085 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53c3ed25-3b0b-46b0-a5ba-1b0b87900743-agent-certs\") pod \"konnectivity-agent-zvjp5\" (UID: \"53c3ed25-3b0b-46b0-a5ba-1b0b87900743\") " pod="kube-system/konnectivity-agent-zvjp5" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974112 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-log-socket\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974139 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f94eece0-4626-4e4e-8c9f-d8aab764fe95-kubelet-config\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974165 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-etc-kubernetes\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974192 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974219 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974222 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fa70596-1837-4f06-b94c-88558d60b65e-ovnkube-config\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974259 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f94eece0-4626-4e4e-8c9f-d8aab764fe95-kubelet-config\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edab0343-4377-4ac0-8106-b39733beefc7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974265 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974304 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/820cce87-a018-4751-a835-a045e7ccfd8e-hosts-file\") pod \"node-resolver-ssx5d\" (UID: \"820cce87-a018-4751-a835-a045e7ccfd8e\") " pod="openshift-dns/node-resolver-ssx5d" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974328 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974330 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-multus-socket-dir-parent\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974396 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-run-k8s-cni-cncf-io\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.975171 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974417 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edab0343-4377-4ac0-8106-b39733beefc7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974427 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-run-systemd\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974457 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edab0343-4377-4ac0-8106-b39733beefc7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974478 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-run-systemd\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974484 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820cce87-a018-4751-a835-a045e7ccfd8e-tmp-dir\") pod \"node-resolver-ssx5d\" (UID: \"820cce87-a018-4751-a835-a045e7ccfd8e\") " pod="openshift-dns/node-resolver-ssx5d" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974508 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-multus-cni-dir\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974226 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-log-socket\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974540 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-socket-dir\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974572 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53c3ed25-3b0b-46b0-a5ba-1b0b87900743-konnectivity-ca\") pod \"konnectivity-agent-zvjp5\" (UID: \"53c3ed25-3b0b-46b0-a5ba-1b0b87900743\") " pod="kube-system/konnectivity-agent-zvjp5" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974597 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50570df0-9213-4039-ab5c-a1347dc88854-host\") pod \"node-ca-rz68t\" (UID: \"50570df0-9213-4039-ab5c-a1347dc88854\") " pod="openshift-image-registry/node-ca-rz68t" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974620 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-run-netns\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974645 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-socket-dir\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974655 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-run-ovn\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974685 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-run-ovn\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974710 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f94eece0-4626-4e4e-8c9f-d8aab764fe95-dbus\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974719 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50570df0-9213-4039-ab5c-a1347dc88854-host\") pod \"node-ca-rz68t\" (UID: \"50570df0-9213-4039-ab5c-a1347dc88854\") " pod="openshift-image-registry/node-ca-rz68t" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edab0343-4377-4ac0-8106-b39733beefc7-cnibin\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.976045 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974752 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-run-netns\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974767 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edab0343-4377-4ac0-8106-b39733beefc7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974795 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-cni-bin\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974819 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-slash\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974846 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d5c6a0a-a492-4aa6-b28b-ce5da66c8079-host-slash\") pod \"iptables-alerter-hnqjc\" (UID: \"5d5c6a0a-a492-4aa6-b28b-ce5da66c8079\") " pod="openshift-network-operator/iptables-alerter-hnqjc" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974865 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edab0343-4377-4ac0-8106-b39733beefc7-cnibin\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974872 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-var-lib-kubelet\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974900 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-hostroot\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974908 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f94eece0-4626-4e4e-8c9f-d8aab764fe95-dbus\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974927 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-modprobe-d\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974917 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-slash\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974970 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-run\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.974982 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-host-cni-bin\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975006 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975032 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d5c6a0a-a492-4aa6-b28b-ce5da66c8079-iptables-alerter-script\") pod \"iptables-alerter-hnqjc\" (UID: \"5d5c6a0a-a492-4aa6-b28b-ce5da66c8079\") " pod="openshift-network-operator/iptables-alerter-hnqjc" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975040 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d5c6a0a-a492-4aa6-b28b-ce5da66c8079-host-slash\") pod \"iptables-alerter-hnqjc\" (UID: \"5d5c6a0a-a492-4aa6-b28b-ce5da66c8079\") " pod="openshift-network-operator/iptables-alerter-hnqjc" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975058 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qh2d\" (UniqueName: \"kubernetes.io/projected/842be287-d8ca-4bf4-9974-091b54c7191c-kube-api-access-9qh2d\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.976814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975089 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-systemd-units\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975114 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-var-lib-openvswitch\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975143 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/edab0343-4377-4ac0-8106-b39733beefc7-os-release\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975198 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-os-release\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975224 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-sysctl-conf\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975252 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/842be287-d8ca-4bf4-9974-091b54c7191c-etc-tuned\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975281 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edab0343-4377-4ac0-8106-b39733beefc7-cni-binary-copy\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975307 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-var-lib-cni-multus\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975331 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-sys\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975331 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edab0343-4377-4ac0-8106-b39733beefc7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975353 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fa70596-1837-4f06-b94c-88558d60b65e-ovnkube-config\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975383 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-registration-dir\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975412 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/50570df0-9213-4039-ab5c-a1347dc88854-serviceca\") pod \"node-ca-rz68t\" (UID: \"50570df0-9213-4039-ab5c-a1347dc88854\") " pod="openshift-image-registry/node-ca-rz68t" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975416 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-systemd-units\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975429 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edab0343-4377-4ac0-8106-b39733beefc7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975472 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53c3ed25-3b0b-46b0-a5ba-1b0b87900743-konnectivity-ca\") pod \"konnectivity-agent-zvjp5\" (UID: \"53c3ed25-3b0b-46b0-a5ba-1b0b87900743\") " pod="kube-system/konnectivity-agent-zvjp5" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975486 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/edab0343-4377-4ac0-8106-b39733beefc7-os-release\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.977667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975433 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fa70596-1837-4f06-b94c-88558d60b65e-var-lib-openvswitch\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.978565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99mm6\" (UniqueName: \"kubernetes.io/projected/50570df0-9213-4039-ab5c-a1347dc88854-kube-api-access-99mm6\") pod \"node-ca-rz68t\" (UID: \"50570df0-9213-4039-ab5c-a1347dc88854\") " pod="openshift-image-registry/node-ca-rz68t" Jul 23 01:28:09.978565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975511 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-registration-dir\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.978565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975609 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d5c6a0a-a492-4aa6-b28b-ce5da66c8079-iptables-alerter-script\") pod \"iptables-alerter-hnqjc\" (UID: \"5d5c6a0a-a492-4aa6-b28b-ce5da66c8079\") " pod="openshift-network-operator/iptables-alerter-hnqjc" Jul 23 01:28:09.978565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975611 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fa70596-1837-4f06-b94c-88558d60b65e-env-overrides\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.978565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975674 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.978565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l42b\" (UniqueName: \"kubernetes.io/projected/2fa70596-1837-4f06-b94c-88558d60b65e-kube-api-access-9l42b\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.978565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975771 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a59326e-1357-42d7-96ae-4d6e43dabf24-cni-binary-copy\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.978565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975802 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-run-netns\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.978565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.975828 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-multus-conf-dir\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:09.978565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.976035 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fa70596-1837-4f06-b94c-88558d60b65e-env-overrides\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.978565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.976055 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/50570df0-9213-4039-ab5c-a1347dc88854-serviceca\") pod \"node-ca-rz68t\" (UID: \"50570df0-9213-4039-ab5c-a1347dc88854\") " pod="openshift-image-registry/node-ca-rz68t" Jul 23 01:28:09.978565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.976061 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edab0343-4377-4ac0-8106-b39733beefc7-cni-binary-copy\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:09.979970 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.979878 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fa70596-1837-4f06-b94c-88558d60b65e-ovn-node-metrics-cert\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.980096 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.980008 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53c3ed25-3b0b-46b0-a5ba-1b0b87900743-agent-certs\") pod \"konnectivity-agent-zvjp5\" (UID: \"53c3ed25-3b0b-46b0-a5ba-1b0b87900743\") " pod="kube-system/konnectivity-agent-zvjp5" Jul 23 01:28:09.987021 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.986952 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlk99\" (UniqueName: \"kubernetes.io/projected/b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12-kube-api-access-xlk99\") pod \"aws-ebs-csi-driver-node-c9bzg\" (UID: \"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:09.988257 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.988175 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l42b\" (UniqueName: \"kubernetes.io/projected/2fa70596-1837-4f06-b94c-88558d60b65e-kube-api-access-9l42b\") pod \"ovnkube-node-f6v4s\" (UID: \"2fa70596-1837-4f06-b94c-88558d60b65e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:09.988257 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.988195 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99mm6\" (UniqueName: \"kubernetes.io/projected/50570df0-9213-4039-ab5c-a1347dc88854-kube-api-access-99mm6\") pod \"node-ca-rz68t\" (UID: \"50570df0-9213-4039-ab5c-a1347dc88854\") " pod="openshift-image-registry/node-ca-rz68t" Jul 23 01:28:09.988460 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.988338 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7pq4\" (UniqueName: \"kubernetes.io/projected/5d5c6a0a-a492-4aa6-b28b-ce5da66c8079-kube-api-access-p7pq4\") pod \"iptables-alerter-hnqjc\" (UID: \"5d5c6a0a-a492-4aa6-b28b-ce5da66c8079\") " pod="openshift-network-operator/iptables-alerter-hnqjc" Jul 23 01:28:09.989669 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:09.989642 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hglj8\" (UniqueName: \"kubernetes.io/projected/edab0343-4377-4ac0-8106-b39733beefc7-kube-api-access-hglj8\") pod \"multus-additional-cni-plugins-xfprg\" (UID: \"edab0343-4377-4ac0-8106-b39733beefc7\") " pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:10.020556 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.020523 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n2ssc" Jul 23 01:28:10.026275 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.026246 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n2ssc" Jul 23 01:28:10.076616 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076560 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-run-netns\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.076616 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076610 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-multus-conf-dir\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076640 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-systemd\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076684 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-lib-modules\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076706 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-run-netns\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076708 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-multus-conf-dir\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076709 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-var-lib-kubelet\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076767 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-var-lib-kubelet\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076774 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-systemd\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-system-cni-dir\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076822 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-system-cni-dir\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076832 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a59326e-1357-42d7-96ae-4d6e43dabf24-multus-daemon-config\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076846 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-lib-modules\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-host\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076893 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbj8q\" (UniqueName: \"kubernetes.io/projected/820cce87-a018-4751-a835-a045e7ccfd8e-kube-api-access-fbj8q\") pod \"node-resolver-ssx5d\" (UID: \"820cce87-a018-4751-a835-a045e7ccfd8e\") " pod="openshift-dns/node-resolver-ssx5d" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076900 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-host\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.076920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076910 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-sysconfig\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076948 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-kubernetes\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.076952 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-sysconfig\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077005 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-cnibin\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077018 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-kubernetes\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077029 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-run-multus-certs\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077056 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-sysctl-d\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077081 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/842be287-d8ca-4bf4-9974-091b54c7191c-tmp\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077083 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-cnibin\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077119 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077138 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-run-multus-certs\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077154 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-var-lib-cni-bin\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077209 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-var-lib-cni-bin\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.077228 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077250 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-sysctl-d\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077279 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h6wv\" (UniqueName: \"kubernetes.io/projected/2a59326e-1357-42d7-96ae-4d6e43dabf24-kube-api-access-5h6wv\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.077301 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs podName:62b44b95-e27a-42a9-b058-0e0363718750 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:10.577280404 +0000 UTC m=+3.122229575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs") pod "network-metrics-daemon-sgszg" (UID: "62b44b95-e27a-42a9-b058-0e0363718750") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:10.077666 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077333 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmhcr\" (UniqueName: \"kubernetes.io/projected/62b44b95-e27a-42a9-b058-0e0363718750-kube-api-access-kmhcr\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077362 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kxq\" (UniqueName: \"kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq\") pod \"network-check-target-dl8vz\" (UID: \"a20bf778-ada3-487a-b3e2-946cbf2366e2\") " pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-etc-kubernetes\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/820cce87-a018-4751-a835-a045e7ccfd8e-hosts-file\") pod \"node-resolver-ssx5d\" (UID: \"820cce87-a018-4751-a835-a045e7ccfd8e\") " pod="openshift-dns/node-resolver-ssx5d" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077448 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-multus-socket-dir-parent\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077473 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a59326e-1357-42d7-96ae-4d6e43dabf24-multus-daemon-config\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077489 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-etc-kubernetes\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077514 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/820cce87-a018-4751-a835-a045e7ccfd8e-hosts-file\") pod \"node-resolver-ssx5d\" (UID: \"820cce87-a018-4751-a835-a045e7ccfd8e\") " pod="openshift-dns/node-resolver-ssx5d" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077526 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-run-k8s-cni-cncf-io\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077563 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820cce87-a018-4751-a835-a045e7ccfd8e-tmp-dir\") pod \"node-resolver-ssx5d\" (UID: \"820cce87-a018-4751-a835-a045e7ccfd8e\") " pod="openshift-dns/node-resolver-ssx5d" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077591 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-multus-cni-dir\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077621 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-run-k8s-cni-cncf-io\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077629 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-var-lib-kubelet\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077661 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-multus-socket-dir-parent\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077673 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-var-lib-kubelet\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077709 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-multus-cni-dir\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077738 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-hostroot\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077762 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-modprobe-d\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.078488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077791 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-run\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077817 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qh2d\" (UniqueName: \"kubernetes.io/projected/842be287-d8ca-4bf4-9974-091b54c7191c-kube-api-access-9qh2d\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-os-release\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077858 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-sysctl-conf\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077875 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/842be287-d8ca-4bf4-9974-091b54c7191c-etc-tuned\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077892 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-var-lib-cni-multus\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077897 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-run\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077908 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-sys\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077919 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-modprobe-d\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077943 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-sys\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077946 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a59326e-1357-42d7-96ae-4d6e43dabf24-cni-binary-copy\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077969 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-host-var-lib-cni-multus\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.077922 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820cce87-a018-4751-a835-a045e7ccfd8e-tmp-dir\") pod \"node-resolver-ssx5d\" (UID: \"820cce87-a018-4751-a835-a045e7ccfd8e\") " pod="openshift-dns/node-resolver-ssx5d" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.078046 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-hostroot\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.078063 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/842be287-d8ca-4bf4-9974-091b54c7191c-etc-sysctl-conf\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.078075 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a59326e-1357-42d7-96ae-4d6e43dabf24-os-release\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.079245 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.079136 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a59326e-1357-42d7-96ae-4d6e43dabf24-cni-binary-copy\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.079965 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.079832 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/842be287-d8ca-4bf4-9974-091b54c7191c-tmp\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.080391 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.080364 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/842be287-d8ca-4bf4-9974-091b54c7191c-etc-tuned\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.085706 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.085680 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbj8q\" (UniqueName: \"kubernetes.io/projected/820cce87-a018-4751-a835-a045e7ccfd8e-kube-api-access-fbj8q\") pod \"node-resolver-ssx5d\" (UID: \"820cce87-a018-4751-a835-a045e7ccfd8e\") " pod="openshift-dns/node-resolver-ssx5d" Jul 23 01:28:10.085837 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.085720 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h6wv\" (UniqueName: \"kubernetes.io/projected/2a59326e-1357-42d7-96ae-4d6e43dabf24-kube-api-access-5h6wv\") pod \"multus-j5hcq\" (UID: \"2a59326e-1357-42d7-96ae-4d6e43dabf24\") " pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.089949 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.089924 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qh2d\" (UniqueName: \"kubernetes.io/projected/842be287-d8ca-4bf4-9974-091b54c7191c-kube-api-access-9qh2d\") pod \"tuned-srw7p\" (UID: \"842be287-d8ca-4bf4-9974-091b54c7191c\") " pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.090855 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.090834 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:28:10.090961 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.090860 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:28:10.090961 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.090874 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m8kxq for pod openshift-network-diagnostics/network-check-target-dl8vz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:10.090961 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.090953 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq podName:a20bf778-ada3-487a-b3e2-946cbf2366e2 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:10.590934803 +0000 UTC m=+3.135883962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m8kxq" (UniqueName: "kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq") pod "network-check-target-dl8vz" (UID: "a20bf778-ada3-487a-b3e2-946cbf2366e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:10.093549 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.093529 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmhcr\" (UniqueName: \"kubernetes.io/projected/62b44b95-e27a-42a9-b058-0e0363718750-kube-api-access-kmhcr\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:10.166533 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.166428 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" Jul 23 01:28:10.179347 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.179319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hnqjc" Jul 23 01:28:10.192192 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.192155 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zvjp5" Jul 23 01:28:10.199844 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.199819 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xfprg" Jul 23 01:28:10.207667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.207618 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rz68t" Jul 23 01:28:10.216647 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.216609 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:10.225610 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.225586 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ssx5d" Jul 23 01:28:10.233687 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.233661 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5hcq" Jul 23 01:28:10.239788 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.239764 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-srw7p" Jul 23 01:28:10.388956 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.388915 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 01:28:10.482656 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.482552 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:10.482847 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.482733 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:10.482847 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.482806 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret podName:f94eece0-4626-4e4e-8c9f-d8aab764fe95 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:11.482787584 +0000 UTC m=+4.027736743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret") pod "global-pull-secret-syncer-vvpsg" (UID: "f94eece0-4626-4e4e-8c9f-d8aab764fe95") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:10.583326 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.583297 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:10.583489 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.583464 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:10.583548 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.583535 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs podName:62b44b95-e27a-42a9-b058-0e0363718750 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:11.583517843 +0000 UTC m=+4.128466998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs") pod "network-metrics-daemon-sgszg" (UID: "62b44b95-e27a-42a9-b058-0e0363718750") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:10.684390 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.684354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kxq\" (UniqueName: \"kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq\") pod \"network-check-target-dl8vz\" (UID: \"a20bf778-ada3-487a-b3e2-946cbf2366e2\") " pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:10.684587 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.684492 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:28:10.684587 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.684508 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:28:10.684587 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.684517 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m8kxq for pod openshift-network-diagnostics/network-check-target-dl8vz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:10.684587 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.684581 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq podName:a20bf778-ada3-487a-b3e2-946cbf2366e2 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:11.684564211 +0000 UTC m=+4.229513382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8kxq" (UniqueName: "kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq") pod "network-check-target-dl8vz" (UID: "a20bf778-ada3-487a-b3e2-946cbf2366e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:10.922585 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:10.922484 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820cce87_a018_4751_a835_a045e7ccfd8e.slice/crio-8e49b70e721ca4ce0df7ebc517b828606bfa58864a63559f2ba306437a8b14ee WatchSource:0}: Error finding container 8e49b70e721ca4ce0df7ebc517b828606bfa58864a63559f2ba306437a8b14ee: Status 404 returned error can't find the container with id 8e49b70e721ca4ce0df7ebc517b828606bfa58864a63559f2ba306437a8b14ee Jul 23 01:28:10.946981 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:10.946957 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c3ed25_3b0b_46b0_a5ba_1b0b87900743.slice/crio-e4464be012f02aee6f5afb64ea296faa6e4d88d51e309a15b3562a2643063924 WatchSource:0}: Error finding container e4464be012f02aee6f5afb64ea296faa6e4d88d51e309a15b3562a2643063924: Status 404 returned error can't find the container with id e4464be012f02aee6f5afb64ea296faa6e4d88d51e309a15b3562a2643063924 Jul 23 01:28:10.947559 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:10.947526 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d24adb_81c0_4b8c_8ecb_eb34bc80ad12.slice/crio-a51b9cd0b918f85cc2a5d8826e73d990407d1a8da5a17b5e5e7119ba72680c74 WatchSource:0}: Error finding container a51b9cd0b918f85cc2a5d8826e73d990407d1a8da5a17b5e5e7119ba72680c74: Status 404 returned error can't find the container with id a51b9cd0b918f85cc2a5d8826e73d990407d1a8da5a17b5e5e7119ba72680c74 Jul 23 01:28:10.948391 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:10.948368 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50570df0_9213_4039_ab5c_a1347dc88854.slice/crio-c9c4f0842025f502924643074f9e79a576c21de63c85bd328c8ad1d85d38bc02 WatchSource:0}: Error finding container c9c4f0842025f502924643074f9e79a576c21de63c85bd328c8ad1d85d38bc02: Status 404 returned error can't find the container with id c9c4f0842025f502924643074f9e79a576c21de63c85bd328c8ad1d85d38bc02 Jul 23 01:28:10.949552 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:10.949525 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5c6a0a_a492_4aa6_b28b_ce5da66c8079.slice/crio-ef6ea9d763d9dd61fa6711be9ae52aa01858fcfcecf7c0568f9f7d41030857b6 WatchSource:0}: Error finding container ef6ea9d763d9dd61fa6711be9ae52aa01858fcfcecf7c0568f9f7d41030857b6: Status 404 returned error can't find the container with id ef6ea9d763d9dd61fa6711be9ae52aa01858fcfcecf7c0568f9f7d41030857b6 Jul 23 01:28:10.950261 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:10.950222 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa70596_1837_4f06_b94c_88558d60b65e.slice/crio-7f185b18ac5478db4a6578fcfc8a91a9abbce4377a48619465cce3d73f27f261 WatchSource:0}: Error finding container 7f185b18ac5478db4a6578fcfc8a91a9abbce4377a48619465cce3d73f27f261: Status 404 returned error can't find the container with id 7f185b18ac5478db4a6578fcfc8a91a9abbce4377a48619465cce3d73f27f261 Jul 23 01:28:10.951535 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:10.951379 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842be287_d8ca_4bf4_9974_091b54c7191c.slice/crio-f8ecb95f798a6ca6b1842bd0eae15cc2d26662ca30b248bcd352f60ce234bd38 WatchSource:0}: Error finding container f8ecb95f798a6ca6b1842bd0eae15cc2d26662ca30b248bcd352f60ce234bd38: Status 404 returned error can't find the container with id f8ecb95f798a6ca6b1842bd0eae15cc2d26662ca30b248bcd352f60ce234bd38 Jul 23 01:28:10.953685 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:10.953662 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedab0343_4377_4ac0_8106_b39733beefc7.slice/crio-626d26fcfe2b2d825d70a03db572833e30722f31f3bd340c441ccb5e220fb953 WatchSource:0}: Error finding container 626d26fcfe2b2d825d70a03db572833e30722f31f3bd340c441ccb5e220fb953: Status 404 returned error can't find the container with id 626d26fcfe2b2d825d70a03db572833e30722f31f3bd340c441ccb5e220fb953 Jul 23 01:28:10.954691 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:10.954647 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a59326e_1357_42d7_96ae_4d6e43dabf24.slice/crio-03596eec20aeb1ba4970aee5b4bb8b284f81128188da853b04a709913c3023d5 WatchSource:0}: Error finding container 03596eec20aeb1ba4970aee5b4bb8b284f81128188da853b04a709913c3023d5: Status 404 returned error can't find the container with id 03596eec20aeb1ba4970aee5b4bb8b284f81128188da853b04a709913c3023d5 Jul 23 01:28:10.969312 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.969288 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:10.969414 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:10.969390 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:10.982821 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.982786 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-srw7p" event={"ID":"842be287-d8ca-4bf4-9974-091b54c7191c","Type":"ContainerStarted","Data":"f8ecb95f798a6ca6b1842bd0eae15cc2d26662ca30b248bcd352f60ce234bd38"} Jul 23 01:28:10.983908 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.983881 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rz68t" event={"ID":"50570df0-9213-4039-ab5c-a1347dc88854","Type":"ContainerStarted","Data":"c9c4f0842025f502924643074f9e79a576c21de63c85bd328c8ad1d85d38bc02"} Jul 23 01:28:10.984988 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.984965 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zvjp5" event={"ID":"53c3ed25-3b0b-46b0-a5ba-1b0b87900743","Type":"ContainerStarted","Data":"e4464be012f02aee6f5afb64ea296faa6e4d88d51e309a15b3562a2643063924"} Jul 23 01:28:10.986122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.986097 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5hcq" event={"ID":"2a59326e-1357-42d7-96ae-4d6e43dabf24","Type":"ContainerStarted","Data":"03596eec20aeb1ba4970aee5b4bb8b284f81128188da853b04a709913c3023d5"} Jul 23 01:28:10.987453 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.987432 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfprg" event={"ID":"edab0343-4377-4ac0-8106-b39733beefc7","Type":"ContainerStarted","Data":"626d26fcfe2b2d825d70a03db572833e30722f31f3bd340c441ccb5e220fb953"} Jul 23 01:28:10.988578 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.988543 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" event={"ID":"2fa70596-1837-4f06-b94c-88558d60b65e","Type":"ContainerStarted","Data":"7f185b18ac5478db4a6578fcfc8a91a9abbce4377a48619465cce3d73f27f261"} Jul 23 01:28:10.989685 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.989661 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hnqjc" event={"ID":"5d5c6a0a-a492-4aa6-b28b-ce5da66c8079","Type":"ContainerStarted","Data":"ef6ea9d763d9dd61fa6711be9ae52aa01858fcfcecf7c0568f9f7d41030857b6"} Jul 23 01:28:10.990696 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.990663 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" event={"ID":"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12","Type":"ContainerStarted","Data":"a51b9cd0b918f85cc2a5d8826e73d990407d1a8da5a17b5e5e7119ba72680c74"} Jul 23 01:28:10.991793 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:10.991773 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ssx5d" event={"ID":"820cce87-a018-4751-a835-a045e7ccfd8e","Type":"ContainerStarted","Data":"8e49b70e721ca4ce0df7ebc517b828606bfa58864a63559f2ba306437a8b14ee"} Jul 23 01:28:11.029976 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:11.029940 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 01:23:10 +0000 UTC" deadline="2028-01-05 18:09:10.802709635 +0000 UTC" Jul 23 01:28:11.029976 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:11.029974 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12760h40m59.772740723s" Jul 23 01:28:11.490517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:11.490424 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:11.490807 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:11.490628 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:11.490807 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:11.490715 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret podName:f94eece0-4626-4e4e-8c9f-d8aab764fe95 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:13.490692855 +0000 UTC m=+6.035642057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret") pod "global-pull-secret-syncer-vvpsg" (UID: "f94eece0-4626-4e4e-8c9f-d8aab764fe95") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:11.591489 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:11.591451 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:11.591689 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:11.591656 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:11.591786 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:11.591749 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs podName:62b44b95-e27a-42a9-b058-0e0363718750 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:13.591727181 +0000 UTC m=+6.136676343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs") pod "network-metrics-daemon-sgszg" (UID: "62b44b95-e27a-42a9-b058-0e0363718750") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:11.692394 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:11.692351 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kxq\" (UniqueName: \"kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq\") pod \"network-check-target-dl8vz\" (UID: \"a20bf778-ada3-487a-b3e2-946cbf2366e2\") " pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:11.692584 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:11.692563 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:28:11.692704 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:11.692594 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:28:11.692704 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:11.692610 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m8kxq for pod openshift-network-diagnostics/network-check-target-dl8vz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:11.692704 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:11.692697 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq podName:a20bf778-ada3-487a-b3e2-946cbf2366e2 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:13.692654345 +0000 UTC m=+6.237603517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8kxq" (UniqueName: "kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq") pod "network-check-target-dl8vz" (UID: "a20bf778-ada3-487a-b3e2-946cbf2366e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:11.971382 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:11.970842 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:11.971382 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:11.970997 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:11.971382 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:11.971054 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:11.971382 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:11.971193 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:12.016742 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:12.016686 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.ec2.internal" event={"ID":"09f69a889a832c91f7f75eb1f3452d24","Type":"ContainerStarted","Data":"29c918b22c2ae99edafb15399b73076b322cd4cf000aedeb6c3e6d8d82edc004"} Jul 23 01:28:12.030753 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:12.030680 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 01:23:10 +0000 UTC" deadline="2028-02-01 18:00:28.273506445 +0000 UTC" Jul 23 01:28:12.030753 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:12.030715 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13408h32m16.24279634s" Jul 23 01:28:12.045333 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:12.044472 2572 generic.go:356] "Generic (PLEG): container finished" podID="d368fcab28cf50f84a474490a3004fef" containerID="2635fb1da860fc3997ef7c80cd793a6710acddb89d4af0296a01545f75e2a25d" exitCode=0 Jul 23 01:28:12.045333 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:12.044525 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" event={"ID":"d368fcab28cf50f84a474490a3004fef","Type":"ContainerDied","Data":"2635fb1da860fc3997ef7c80cd793a6710acddb89d4af0296a01545f75e2a25d"} Jul 23 01:28:12.064237 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:12.064177 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.ec2.internal" podStartSLOduration=2.064154122 podStartE2EDuration="2.064154122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:12.035493024 +0000 UTC m=+4.580442202" watchObservedRunningTime="2026-07-23 01:28:12.064154122 +0000 UTC m=+4.609103313" Jul 23 01:28:12.969313 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:12.969222 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:12.969492 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:12.969421 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:13.063894 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:13.063806 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" event={"ID":"d368fcab28cf50f84a474490a3004fef","Type":"ContainerStarted","Data":"7f7e12ca6a0d80b93c2c8a7b1fef808ef5294d3f2231a9e544565070372792df"} Jul 23 01:28:13.506443 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:13.506401 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:13.506627 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:13.506565 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:13.506734 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:13.506631 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret podName:f94eece0-4626-4e4e-8c9f-d8aab764fe95 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:17.50661177 +0000 UTC m=+10.051560941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret") pod "global-pull-secret-syncer-vvpsg" (UID: "f94eece0-4626-4e4e-8c9f-d8aab764fe95") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:13.609951 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:13.609910 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:13.610145 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:13.610103 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:13.610212 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:13.610170 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs podName:62b44b95-e27a-42a9-b058-0e0363718750 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:17.610150335 +0000 UTC m=+10.155099488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs") pod "network-metrics-daemon-sgszg" (UID: "62b44b95-e27a-42a9-b058-0e0363718750") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:13.711986 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:13.711303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kxq\" (UniqueName: \"kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq\") pod \"network-check-target-dl8vz\" (UID: \"a20bf778-ada3-487a-b3e2-946cbf2366e2\") " pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:13.711986 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:13.711490 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:28:13.711986 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:13.711513 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:28:13.711986 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:13.711543 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m8kxq for pod openshift-network-diagnostics/network-check-target-dl8vz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:13.711986 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:13.711605 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq podName:a20bf778-ada3-487a-b3e2-946cbf2366e2 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:17.711585497 +0000 UTC m=+10.256534654 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8kxq" (UniqueName: "kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq") pod "network-check-target-dl8vz" (UID: "a20bf778-ada3-487a-b3e2-946cbf2366e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:13.981785 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:13.975045 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:13.981785 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:13.977687 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:13.981785 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:13.975259 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:13.983346 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:13.983314 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:14.969737 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:14.969703 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:14.970209 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:14.969834 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:15.970058 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:15.970020 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:15.970677 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:15.970189 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:15.970677 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:15.970625 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:15.970855 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:15.970751 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:16.969707 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:16.969193 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:16.969707 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:16.969337 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:17.547484 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:17.547445 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:17.548012 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:17.547723 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:17.548012 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:17.547800 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret podName:f94eece0-4626-4e4e-8c9f-d8aab764fe95 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:25.547776454 +0000 UTC m=+18.092725613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret") pod "global-pull-secret-syncer-vvpsg" (UID: "f94eece0-4626-4e4e-8c9f-d8aab764fe95") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:17.648548 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:17.648315 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:17.648548 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:17.648515 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:17.648832 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:17.648591 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs podName:62b44b95-e27a-42a9-b058-0e0363718750 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:25.648570552 +0000 UTC m=+18.193519710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs") pod "network-metrics-daemon-sgszg" (UID: "62b44b95-e27a-42a9-b058-0e0363718750") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:17.749910 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:17.749858 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kxq\" (UniqueName: \"kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq\") pod \"network-check-target-dl8vz\" (UID: \"a20bf778-ada3-487a-b3e2-946cbf2366e2\") " pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:17.750089 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:17.750029 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:28:17.750089 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:17.750055 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:28:17.750089 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:17.750068 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m8kxq for pod openshift-network-diagnostics/network-check-target-dl8vz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:17.750267 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:17.750138 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq podName:a20bf778-ada3-487a-b3e2-946cbf2366e2 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:25.750115069 +0000 UTC m=+18.295064257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8kxq" (UniqueName: "kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq") pod "network-check-target-dl8vz" (UID: "a20bf778-ada3-487a-b3e2-946cbf2366e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:17.971536 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:17.970825 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:17.971536 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:17.970966 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:17.971536 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:17.971339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:17.971536 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:17.971441 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:18.970374 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:18.970171 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:18.970374 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:18.970324 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:19.970117 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:19.970004 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:19.970319 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:19.970158 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:19.970696 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:19.970483 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:19.970696 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:19.970622 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:20.970221 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:20.970183 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:20.970404 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:20.970313 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:21.972539 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:21.972507 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:21.972539 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:21.972547 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:21.973021 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:21.972629 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:21.973021 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:21.972750 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:22.970222 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:22.970187 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:22.970410 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:22.970297 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:23.969934 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:23.969835 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:23.969934 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:23.969893 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:23.970389 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:23.969995 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:23.970389 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:23.970116 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:24.969221 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:24.969181 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:24.969429 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:24.969317 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:25.618514 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:25.618484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:25.618967 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:25.618687 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:25.618967 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:25.618762 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret podName:f94eece0-4626-4e4e-8c9f-d8aab764fe95 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:41.618743329 +0000 UTC m=+34.163692485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret") pod "global-pull-secret-syncer-vvpsg" (UID: "f94eece0-4626-4e4e-8c9f-d8aab764fe95") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:25.719123 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:25.719080 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:25.719303 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:25.719253 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:25.719358 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:25.719329 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs podName:62b44b95-e27a-42a9-b058-0e0363718750 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:41.719309082 +0000 UTC m=+34.264258235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs") pod "network-metrics-daemon-sgszg" (UID: "62b44b95-e27a-42a9-b058-0e0363718750") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:25.819847 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:25.819813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kxq\" (UniqueName: \"kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq\") pod \"network-check-target-dl8vz\" (UID: \"a20bf778-ada3-487a-b3e2-946cbf2366e2\") " pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:25.820010 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:25.819958 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:28:25.820010 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:25.819976 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:28:25.820010 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:25.819991 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m8kxq for pod openshift-network-diagnostics/network-check-target-dl8vz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:25.820156 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:25.820049 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq podName:a20bf778-ada3-487a-b3e2-946cbf2366e2 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:41.82003425 +0000 UTC m=+34.364983401 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8kxq" (UniqueName: "kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq") pod "network-check-target-dl8vz" (UID: "a20bf778-ada3-487a-b3e2-946cbf2366e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:25.969361 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:25.969339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:25.969463 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:25.969441 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:25.969530 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:25.969515 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:25.969620 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:25.969605 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:26.969303 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:26.969265 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:26.969719 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:26.969375 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:27.970560 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:27.970526 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:27.971048 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:27.970720 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:27.971048 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:27.970793 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:27.971048 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:27.970913 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:28.969147 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:28.969118 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:28.969323 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:28.969230 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:29.969446 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:29.969409 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:29.970036 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:29.969409 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:29.970036 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:29.969570 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:29.970036 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:29.969722 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:30.970319 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:30.969914 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:30.970952 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:30.970437 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:31.108457 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.108370 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" event={"ID":"2fa70596-1837-4f06-b94c-88558d60b65e","Type":"ContainerStarted","Data":"f77760cc60cd2a897c8f90c33a385d6d856a081fe38d3a9935cd3ce8acdace58"} Jul 23 01:28:31.108614 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.108493 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" event={"ID":"2fa70596-1837-4f06-b94c-88558d60b65e","Type":"ContainerStarted","Data":"ba2e8e261b64a1c8e9d4b1bbd2ee0e44bc5de461f5a8dea41be891db6e21b9d8"} Jul 23 01:28:31.108614 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.108548 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" event={"ID":"2fa70596-1837-4f06-b94c-88558d60b65e","Type":"ContainerStarted","Data":"b363ed121c331ba7e86a9993e6abf604e3b4ea9bdc15a1ade0e94a5d3b6e365c"} Jul 23 01:28:31.108755 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.108656 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" event={"ID":"2fa70596-1837-4f06-b94c-88558d60b65e","Type":"ContainerStarted","Data":"7ba887de8a382f42c9a9b384adcb3ee9e34f1f816ee127c62f928cbdeb147966"} Jul 23 01:28:31.108755 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.108698 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" event={"ID":"2fa70596-1837-4f06-b94c-88558d60b65e","Type":"ContainerStarted","Data":"8c67b0795acd6cde727b2f9455227c130b142e0da1f3687772104b5972e4463f"} Jul 23 01:28:31.108755 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.108741 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" event={"ID":"2fa70596-1837-4f06-b94c-88558d60b65e","Type":"ContainerStarted","Data":"44e2b46c26e5f9bc9e13e10b6033a03a2c26a9946b85ab68f4e8be340d1c215f"} Jul 23 01:28:31.109925 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.109896 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" event={"ID":"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12","Type":"ContainerStarted","Data":"0cdb462f1b6bb8d3cd8bde1fb2df211fd445eb23f82875124aeae85f2624de3b"} Jul 23 01:28:31.111571 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.111543 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ssx5d" event={"ID":"820cce87-a018-4751-a835-a045e7ccfd8e","Type":"ContainerStarted","Data":"82f7a496eb948c8e73a3189d8aaae9d1ee0deb539f734cdb8c00f88cd55c1fdc"} Jul 23 01:28:31.113237 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.113189 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-srw7p" event={"ID":"842be287-d8ca-4bf4-9974-091b54c7191c","Type":"ContainerStarted","Data":"081fd57ece855ffe08c7afbbf8483e18c1d6e7dfeadfb705f2e9d190b9198822"} Jul 23 01:28:31.114963 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.114937 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rz68t" event={"ID":"50570df0-9213-4039-ab5c-a1347dc88854","Type":"ContainerStarted","Data":"251e5a0261dcbb81d16fccab68f182d79a1291b89d5b7cb08cb0c3531d4ed822"} Jul 23 01:28:31.116684 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.116625 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zvjp5" event={"ID":"53c3ed25-3b0b-46b0-a5ba-1b0b87900743","Type":"ContainerStarted","Data":"41d4c910c487a1ee187d363a99631d95aa8064a57391eef1347bc9b81551e7ef"} Jul 23 01:28:31.118276 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.118255 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5hcq" event={"ID":"2a59326e-1357-42d7-96ae-4d6e43dabf24","Type":"ContainerStarted","Data":"032a4b1357673d3a9039c894d14495b039ef80b430df0eab759c602bd2dbcfc3"} Jul 23 01:28:31.120013 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.119992 2572 generic.go:356] "Generic (PLEG): container finished" podID="edab0343-4377-4ac0-8106-b39733beefc7" containerID="0fe69197d3b12bd6ea59666c838b74293e64d83737bcd1298df5592a037bb60d" exitCode=0 Jul 23 01:28:31.120098 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.120025 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfprg" event={"ID":"edab0343-4377-4ac0-8106-b39733beefc7","Type":"ContainerDied","Data":"0fe69197d3b12bd6ea59666c838b74293e64d83737bcd1298df5592a037bb60d"} Jul 23 01:28:31.125555 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.125508 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.ec2.internal" podStartSLOduration=21.125491574 podStartE2EDuration="21.125491574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:13.082185249 +0000 UTC m=+5.627134427" watchObservedRunningTime="2026-07-23 01:28:31.125491574 +0000 UTC m=+23.670440749" Jul 23 01:28:31.126029 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.125995 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ssx5d" podStartSLOduration=2.778356927 podStartE2EDuration="22.125986481s" totalImagesPullingTime="19.347629554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:31.125101141 +0000 UTC m=+23.670050334" watchObservedRunningTime="2026-07-23 01:28:31.125986481 +0000 UTC m=+23.670935654" Jul 23 01:28:31.137888 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.137848 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zvjp5" podStartSLOduration=7.125110836 podStartE2EDuration="22.137833862s" totalImagesPullingTime="15.012723026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:31.137824091 +0000 UTC m=+23.682773284" watchObservedRunningTime="2026-07-23 01:28:31.137833862 +0000 UTC m=+23.682783075" Jul 23 01:28:31.150912 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.150868 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rz68t" podStartSLOduration=2.811435994 podStartE2EDuration="22.150853412s" totalImagesPullingTime="19.339417418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:31.150792237 +0000 UTC m=+23.695741409" watchObservedRunningTime="2026-07-23 01:28:31.150853412 +0000 UTC m=+23.695802562" Jul 23 01:28:31.165824 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.165776 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j5hcq" podStartSLOduration=2.771478769 podStartE2EDuration="22.165760451s" totalImagesPullingTime="19.394281682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:31.165721403 +0000 UTC m=+23.710670587" watchObservedRunningTime="2026-07-23 01:28:31.165760451 +0000 UTC m=+23.710709605" Jul 23 01:28:31.185475 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.185429 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-srw7p" podStartSLOduration=2.84677373 podStartE2EDuration="22.185413744s" totalImagesPullingTime="19.338640014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:31.185391329 +0000 UTC m=+23.730340504" watchObservedRunningTime="2026-07-23 01:28:31.185413744 +0000 UTC m=+23.730362917" Jul 23 01:28:31.774664 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.774618 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 01:28:31.934619 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.934442 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T01:28:31.774656687Z","UUID":"a6e27645-56de-4d8a-b83c-f5d4e794656b","Handler":null,"Name":"","Endpoint":""} Jul 23 01:28:31.935875 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.935855 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 01:28:31.936009 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.935900 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 01:28:31.970253 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.970224 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:31.970439 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:31.970231 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:31.970439 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:31.970370 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:31.970439 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:31.970418 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:32.123839 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:32.123787 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hnqjc" event={"ID":"5d5c6a0a-a492-4aa6-b28b-ce5da66c8079","Type":"ContainerStarted","Data":"af657cc273887a5e4c78266e303fc565185c5c0836f332c99352b9f066a06021"} Jul 23 01:28:32.126028 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:32.125986 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" event={"ID":"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12","Type":"ContainerStarted","Data":"5ef54c9a9bb57d76c2380d4055c72c72a4081bc54d3ee7a4d7f542ec0a3b0e69"} Jul 23 01:28:32.970183 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:32.969967 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:32.970354 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:32.970273 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:33.130649 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:33.130605 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" event={"ID":"b2d24adb-81c0-4b8c-8ecb-eb34bc80ad12","Type":"ContainerStarted","Data":"d03f272c042d3afe274cfdb259a89ca6dee3ce80b6782db169433e63c498fcca"} Jul 23 01:28:33.147091 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:33.147036 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hnqjc" podStartSLOduration=4.807391 podStartE2EDuration="24.147015783s" totalImagesPullingTime="19.339624783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:32.137193146 +0000 UTC m=+24.682142320" watchObservedRunningTime="2026-07-23 01:28:33.147015783 +0000 UTC m=+25.691964958" Jul 23 01:28:33.147572 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:33.147546 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9bzg" podStartSLOduration=2.570030893 podStartE2EDuration="24.14753724s" totalImagesPullingTime="21.577506347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:33.146562743 +0000 UTC m=+25.691511944" watchObservedRunningTime="2026-07-23 01:28:33.14753724 +0000 UTC m=+25.692486413" Jul 23 01:28:33.970088 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:33.970051 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:33.970297 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:33.970202 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:33.970297 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:33.970260 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:33.970420 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:33.970380 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:34.136000 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:34.135965 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" event={"ID":"2fa70596-1837-4f06-b94c-88558d60b65e","Type":"ContainerStarted","Data":"d44f26a1c2966f3e661f6250ad3200b468a6fea6344ef5bf6bd8fd13aaab8b81"} Jul 23 01:28:34.969999 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:34.969965 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:34.970183 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:34.970087 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:35.512878 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:35.512617 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zvjp5" Jul 23 01:28:35.513600 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:35.513584 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zvjp5" Jul 23 01:28:35.969534 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:35.969465 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:35.969690 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:35.969561 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:35.969690 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:35.969596 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:35.969808 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:35.969686 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:36.141481 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:36.141447 2572 generic.go:356] "Generic (PLEG): container finished" podID="edab0343-4377-4ac0-8106-b39733beefc7" containerID="f891a95496c1cbd113070f671de8fd8e5768f2f21fd7e49364a3bd7fa659459e" exitCode=0 Jul 23 01:28:36.141672 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:36.141526 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfprg" event={"ID":"edab0343-4377-4ac0-8106-b39733beefc7","Type":"ContainerDied","Data":"f891a95496c1cbd113070f671de8fd8e5768f2f21fd7e49364a3bd7fa659459e"} Jul 23 01:28:36.145199 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:36.145176 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" event={"ID":"2fa70596-1837-4f06-b94c-88558d60b65e","Type":"ContainerStarted","Data":"bcb38c0197956068e712d679697f5c4f1ef31208f9b74096c977f1928be4277c"} Jul 23 01:28:36.145489 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:36.145466 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zvjp5" Jul 23 01:28:36.146037 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:36.146021 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zvjp5" Jul 23 01:28:36.190252 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:36.190204 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" podStartSLOduration=7.778557331 podStartE2EDuration="27.190190269s" totalImagesPullingTime="19.411632938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:36.189125108 +0000 UTC m=+28.734074280" watchObservedRunningTime="2026-07-23 01:28:36.190190269 +0000 UTC m=+28.735139442" Jul 23 01:28:36.969969 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:36.969932 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:36.970442 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:36.970075 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:37.051818 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:37.051784 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:37.148124 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:37.148088 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:37.148303 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:37.148179 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:37.166656 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:37.166616 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:37.166803 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:37.166787 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:28:37.406918 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:37.406735 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dl8vz"] Jul 23 01:28:37.407062 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:37.407027 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:37.407125 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:37.407107 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:37.407467 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:37.407441 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vvpsg"] Jul 23 01:28:37.407567 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:37.407554 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:37.407670 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:37.407650 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:37.409660 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:37.409627 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sgszg"] Jul 23 01:28:37.409744 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:37.409738 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:37.409832 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:37.409815 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:38.151443 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:38.151412 2572 generic.go:356] "Generic (PLEG): container finished" podID="edab0343-4377-4ac0-8106-b39733beefc7" containerID="af733e8e1604450b927bf4caa1ecf20befd6ffc74235a4ffc7adf82c8f447c5a" exitCode=0 Jul 23 01:28:38.151950 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:38.151502 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfprg" event={"ID":"edab0343-4377-4ac0-8106-b39733beefc7","Type":"ContainerDied","Data":"af733e8e1604450b927bf4caa1ecf20befd6ffc74235a4ffc7adf82c8f447c5a"} Jul 23 01:28:38.970156 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:38.970124 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:38.970314 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:38.970124 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:38.970314 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:38.970239 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:38.970314 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:38.970124 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:38.970314 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:38.970301 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:38.970441 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:38.970383 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:40.158892 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:40.158853 2572 generic.go:356] "Generic (PLEG): container finished" podID="edab0343-4377-4ac0-8106-b39733beefc7" containerID="2350cf8e51b7e2757a953b84a9673ee8fde1f4f86ea12bb6798056a6431a5725" exitCode=0 Jul 23 01:28:40.159470 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:40.158924 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfprg" event={"ID":"edab0343-4377-4ac0-8106-b39733beefc7","Type":"ContainerDied","Data":"2350cf8e51b7e2757a953b84a9673ee8fde1f4f86ea12bb6798056a6431a5725"} Jul 23 01:28:40.970097 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:40.970060 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:40.970291 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:40.970063 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:40.970291 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:40.970172 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:40.970291 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:40.970225 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:40.970291 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:40.970063 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:40.970500 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:40.970375 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:41.645392 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:41.645349 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:41.645891 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:41.645475 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:41.645891 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:41.645554 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret podName:f94eece0-4626-4e4e-8c9f-d8aab764fe95 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:13.645531426 +0000 UTC m=+66.190480597 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret") pod "global-pull-secret-syncer-vvpsg" (UID: "f94eece0-4626-4e4e-8c9f-d8aab764fe95") : object "kube-system"/"original-pull-secret" not registered Jul 23 01:28:41.746009 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:41.745970 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:41.746186 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:41.746126 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:41.746252 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:41.746197 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs podName:62b44b95-e27a-42a9-b058-0e0363718750 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:13.746178243 +0000 UTC m=+66.291127393 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs") pod "network-metrics-daemon-sgszg" (UID: "62b44b95-e27a-42a9-b058-0e0363718750") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 01:28:41.846708 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:41.846652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kxq\" (UniqueName: \"kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq\") pod \"network-check-target-dl8vz\" (UID: \"a20bf778-ada3-487a-b3e2-946cbf2366e2\") " pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:41.846893 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:41.846814 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 01:28:41.846893 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:41.846844 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 01:28:41.846893 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:41.846857 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m8kxq for pod openshift-network-diagnostics/network-check-target-dl8vz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:41.847013 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:41.846923 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq podName:a20bf778-ada3-487a-b3e2-946cbf2366e2 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:13.846903927 +0000 UTC m=+66.391853084 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8kxq" (UniqueName: "kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq") pod "network-check-target-dl8vz" (UID: "a20bf778-ada3-487a-b3e2-946cbf2366e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 01:28:42.970098 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:42.970062 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:42.970696 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:42.970183 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:42.970696 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:42.970195 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:42.970696 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:42.970190 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvpsg" podUID="f94eece0-4626-4e4e-8c9f-d8aab764fe95" Jul 23 01:28:42.970696 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:42.970312 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgszg" podUID="62b44b95-e27a-42a9-b058-0e0363718750" Jul 23 01:28:42.970696 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:42.970397 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dl8vz" podUID="a20bf778-ada3-487a-b3e2-946cbf2366e2" Jul 23 01:28:44.213296 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.213274 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-117.ec2.internal" event="NodeReady" Jul 23 01:28:44.213654 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.213415 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 01:28:44.257382 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.257354 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c4dbbf699-qm7vs"] Jul 23 01:28:44.258967 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.258951 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fdh5s"] Jul 23 01:28:44.259088 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.259069 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.260751 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.260729 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ghjqn"] Jul 23 01:28:44.260895 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.260873 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.261596 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.261569 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h6src\"" Jul 23 01:28:44.261737 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.261580 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 01:28:44.261737 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.261629 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 01:28:44.261737 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.261660 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 01:28:44.262379 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.262357 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ghjqn" Jul 23 01:28:44.263128 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.263100 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 01:28:44.263259 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.263183 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 01:28:44.263259 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.263197 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qjsgq\"" Jul 23 01:28:44.264931 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.264913 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 01:28:44.264931 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.264930 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 01:28:44.265122 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.265108 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 01:28:44.265419 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.265393 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cl24z\"" Jul 23 01:28:44.267912 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.267892 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 01:28:44.270180 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.270151 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fdh5s"] Jul 23 01:28:44.274307 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.274246 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c4dbbf699-qm7vs"] Jul 23 01:28:44.282480 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.282459 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ghjqn"] Jul 23 01:28:44.361855 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.361829 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqsr\" (UniqueName: \"kubernetes.io/projected/3bef66d3-4454-4b73-b6ce-a9f8b034622f-kube-api-access-bmqsr\") pod \"ingress-canary-ghjqn\" (UID: \"3bef66d3-4454-4b73-b6ce-a9f8b034622f\") " pod="openshift-ingress-canary/ingress-canary-ghjqn" Jul 23 01:28:44.361957 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.361860 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.361957 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.361879 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-bound-sa-token\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.361957 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.361897 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bef66d3-4454-4b73-b6ce-a9f8b034622f-cert\") pod \"ingress-canary-ghjqn\" (UID: \"3bef66d3-4454-4b73-b6ce-a9f8b034622f\") " pod="openshift-ingress-canary/ingress-canary-ghjqn" Jul 23 01:28:44.361957 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.361916 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/75178789-2a57-449a-8fed-79136d817399-config-volume\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.361957 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.361932 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-image-registry-private-configuration\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.361957 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.361954 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.362161 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.361972 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g944g\" (UniqueName: \"kubernetes.io/projected/75178789-2a57-449a-8fed-79136d817399-kube-api-access-g944g\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.362161 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.362018 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75178789-2a57-449a-8fed-79136d817399-tmp-dir\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.362161 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.362048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-ca-trust-extracted\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.362161 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.362071 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62rjg\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-kube-api-access-62rjg\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.362161 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.362095 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-certificates\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.362161 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.362117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-installation-pull-secrets\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.362334 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.362169 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-trusted-ca\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.462791 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.462706 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bef66d3-4454-4b73-b6ce-a9f8b034622f-cert\") pod \"ingress-canary-ghjqn\" (UID: \"3bef66d3-4454-4b73-b6ce-a9f8b034622f\") " pod="openshift-ingress-canary/ingress-canary-ghjqn" Jul 23 01:28:44.462791 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.462763 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/75178789-2a57-449a-8fed-79136d817399-config-volume\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.462791 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.462789 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-image-registry-private-configuration\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.463027 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.462825 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.463027 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.462856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g944g\" (UniqueName: \"kubernetes.io/projected/75178789-2a57-449a-8fed-79136d817399-kube-api-access-g944g\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.463027 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.462885 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75178789-2a57-449a-8fed-79136d817399-tmp-dir\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.463027 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.462909 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-ca-trust-extracted\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.463027 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.462936 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62rjg\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-kube-api-access-62rjg\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.463027 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:44.462945 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 01:28:44.463027 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.462976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-certificates\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.463027 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.462998 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-installation-pull-secrets\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.463398 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:44.463038 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls podName:75178789-2a57-449a-8fed-79136d817399 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:44.963011718 +0000 UTC m=+37.507960874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls") pod "dns-default-fdh5s" (UID: "75178789-2a57-449a-8fed-79136d817399") : secret "dns-default-metrics-tls" not found Jul 23 01:28:44.463398 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.463064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-trusted-ca\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.463398 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.463156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqsr\" (UniqueName: \"kubernetes.io/projected/3bef66d3-4454-4b73-b6ce-a9f8b034622f-kube-api-access-bmqsr\") pod \"ingress-canary-ghjqn\" (UID: \"3bef66d3-4454-4b73-b6ce-a9f8b034622f\") " pod="openshift-ingress-canary/ingress-canary-ghjqn" Jul 23 01:28:44.463398 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.463189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.463398 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.463214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-bound-sa-token\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.463398 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.463271 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-ca-trust-extracted\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.465349 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.464458 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-certificates\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.465349 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:44.464569 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 01:28:44.465349 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:44.464673 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4dbbf699-qm7vs: secret "image-registry-tls" not found Jul 23 01:28:44.465349 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:44.464823 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls podName:8cd1b24e-6e1e-40fd-84d9-46e49c57c333 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:44.964717733 +0000 UTC m=+37.509666901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls") pod "image-registry-7c4dbbf699-qm7vs" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333") : secret "image-registry-tls" not found Jul 23 01:28:44.465349 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.464891 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-trusted-ca\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.465349 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.465053 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75178789-2a57-449a-8fed-79136d817399-tmp-dir\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.465349 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.465188 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/75178789-2a57-449a-8fed-79136d817399-config-volume\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.468262 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.467879 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-image-registry-private-configuration\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.469009 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.468988 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-installation-pull-secrets\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.469111 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.469097 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bef66d3-4454-4b73-b6ce-a9f8b034622f-cert\") pod \"ingress-canary-ghjqn\" (UID: \"3bef66d3-4454-4b73-b6ce-a9f8b034622f\") " pod="openshift-ingress-canary/ingress-canary-ghjqn" Jul 23 01:28:44.471337 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.471309 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g944g\" (UniqueName: \"kubernetes.io/projected/75178789-2a57-449a-8fed-79136d817399-kube-api-access-g944g\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.471798 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.471770 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-bound-sa-token\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.471898 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.471881 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqsr\" (UniqueName: \"kubernetes.io/projected/3bef66d3-4454-4b73-b6ce-a9f8b034622f-kube-api-access-bmqsr\") pod \"ingress-canary-ghjqn\" (UID: \"3bef66d3-4454-4b73-b6ce-a9f8b034622f\") " pod="openshift-ingress-canary/ingress-canary-ghjqn" Jul 23 01:28:44.471999 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.471984 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62rjg\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-kube-api-access-62rjg\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.583535 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.583503 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ghjqn" Jul 23 01:28:44.710503 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.710474 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ghjqn"] Jul 23 01:28:44.713784 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:44.713697 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bef66d3_4454_4b73_b6ce_a9f8b034622f.slice/crio-062d1b27ba62b7f4ac52ae63f17618b3fe144c49beb9cbe224108dad2c27aa21 WatchSource:0}: Error finding container 062d1b27ba62b7f4ac52ae63f17618b3fe144c49beb9cbe224108dad2c27aa21: Status 404 returned error can't find the container with id 062d1b27ba62b7f4ac52ae63f17618b3fe144c49beb9cbe224108dad2c27aa21 Jul 23 01:28:44.968249 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.968166 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:44.968377 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:44.968314 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 01:28:44.968377 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.968350 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:44.968377 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:44.968372 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls podName:75178789-2a57-449a-8fed-79136d817399 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:45.968356152 +0000 UTC m=+38.513305303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls") pod "dns-default-fdh5s" (UID: "75178789-2a57-449a-8fed-79136d817399") : secret "dns-default-metrics-tls" not found Jul 23 01:28:44.968484 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:44.968426 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 01:28:44.968484 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:44.968440 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4dbbf699-qm7vs: secret "image-registry-tls" not found Jul 23 01:28:44.968544 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:44.968488 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls podName:8cd1b24e-6e1e-40fd-84d9-46e49c57c333 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:45.968471962 +0000 UTC m=+38.513421124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls") pod "image-registry-7c4dbbf699-qm7vs" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333") : secret "image-registry-tls" not found Jul 23 01:28:44.969173 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.969159 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:28:44.969281 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.969272 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:28:44.969314 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.969287 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:28:44.971859 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.971838 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 01:28:44.972555 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.972522 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 01:28:44.972555 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.972546 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 01:28:44.972881 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.972581 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lg5p2\"" Jul 23 01:28:44.972881 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.972590 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-frx6l\"" Jul 23 01:28:44.972881 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:44.972581 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 01:28:45.174355 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:45.174325 2572 generic.go:356] "Generic (PLEG): container finished" podID="edab0343-4377-4ac0-8106-b39733beefc7" containerID="4cf6e91ad543e46f4c59fdc445c26c0fcf438454dbd651551e36418039abaaf4" exitCode=0 Jul 23 01:28:45.174488 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:45.174404 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfprg" event={"ID":"edab0343-4377-4ac0-8106-b39733beefc7","Type":"ContainerDied","Data":"4cf6e91ad543e46f4c59fdc445c26c0fcf438454dbd651551e36418039abaaf4"} Jul 23 01:28:45.175720 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:45.175492 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ghjqn" event={"ID":"3bef66d3-4454-4b73-b6ce-a9f8b034622f","Type":"ContainerStarted","Data":"062d1b27ba62b7f4ac52ae63f17618b3fe144c49beb9cbe224108dad2c27aa21"} Jul 23 01:28:45.976475 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:45.976249 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:45.976966 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:45.976528 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:45.976966 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:45.976399 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 01:28:45.976966 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:45.976663 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4dbbf699-qm7vs: secret "image-registry-tls" not found Jul 23 01:28:45.976966 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:45.976683 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 01:28:45.976966 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:45.976738 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls podName:8cd1b24e-6e1e-40fd-84d9-46e49c57c333 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:47.976716214 +0000 UTC m=+40.521665368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls") pod "image-registry-7c4dbbf699-qm7vs" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333") : secret "image-registry-tls" not found Jul 23 01:28:45.976966 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:45.976758 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls podName:75178789-2a57-449a-8fed-79136d817399 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:47.976747455 +0000 UTC m=+40.521696605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls") pod "dns-default-fdh5s" (UID: "75178789-2a57-449a-8fed-79136d817399") : secret "dns-default-metrics-tls" not found Jul 23 01:28:46.184240 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:46.184210 2572 generic.go:356] "Generic (PLEG): container finished" podID="edab0343-4377-4ac0-8106-b39733beefc7" containerID="4c6c3661cb326436c92af45854b95320f49cfdd8543dd5872d00ce56c24c7ffb" exitCode=0 Jul 23 01:28:46.184410 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:46.184262 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfprg" event={"ID":"edab0343-4377-4ac0-8106-b39733beefc7","Type":"ContainerDied","Data":"4c6c3661cb326436c92af45854b95320f49cfdd8543dd5872d00ce56c24c7ffb"} Jul 23 01:28:47.188547 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:47.188504 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ghjqn" event={"ID":"3bef66d3-4454-4b73-b6ce-a9f8b034622f","Type":"ContainerStarted","Data":"b0b033825816b7685c34edd917218891ab5afd39c45c3cc8a0a31de7252c2d6e"} Jul 23 01:28:47.192540 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:47.192513 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfprg" event={"ID":"edab0343-4377-4ac0-8106-b39733beefc7","Type":"ContainerStarted","Data":"e657b0b0cf64d61183000e83bb77f5e55ffe275bca0f66de2ce83a06af4474e0"} Jul 23 01:28:47.204123 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:47.204077 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ghjqn" podStartSLOduration=0.871228361 podStartE2EDuration="3.204065101s" totalImagesPullingTime="2.33283674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:47.203454831 +0000 UTC m=+39.748404004" watchObservedRunningTime="2026-07-23 01:28:47.204065101 +0000 UTC m=+39.749014274" Jul 23 01:28:47.225219 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:47.225165 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xfprg" podStartSLOduration=8.610334738 podStartE2EDuration="38.225149934s" totalImagesPullingTime="29.614815196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:47.223673105 +0000 UTC m=+39.768622278" watchObservedRunningTime="2026-07-23 01:28:47.225149934 +0000 UTC m=+39.770099108" Jul 23 01:28:47.990808 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:47.990778 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:47.990983 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:47.990908 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 01:28:47.990983 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:47.990927 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:47.990983 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:47.990961 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls podName:75178789-2a57-449a-8fed-79136d817399 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:51.990945576 +0000 UTC m=+44.535894727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls") pod "dns-default-fdh5s" (UID: "75178789-2a57-449a-8fed-79136d817399") : secret "dns-default-metrics-tls" not found Jul 23 01:28:47.991106 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:47.991004 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 01:28:47.991106 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:47.991014 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4dbbf699-qm7vs: secret "image-registry-tls" not found Jul 23 01:28:47.991106 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:47.991060 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls podName:8cd1b24e-6e1e-40fd-84d9-46e49c57c333 nodeName:}" failed. No retries permitted until 2026-07-23 01:28:51.991049409 +0000 UTC m=+44.535998559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls") pod "image-registry-7c4dbbf699-qm7vs" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333") : secret "image-registry-tls" not found Jul 23 01:28:52.020336 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.020292 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:28:52.020800 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.020409 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:28:52.020800 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:52.020458 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 01:28:52.020800 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:52.020528 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 01:28:52.020800 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:52.020542 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4dbbf699-qm7vs: secret "image-registry-tls" not found Jul 23 01:28:52.020800 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:52.020530 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls podName:75178789-2a57-449a-8fed-79136d817399 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:00.020510235 +0000 UTC m=+52.565459417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls") pod "dns-default-fdh5s" (UID: "75178789-2a57-449a-8fed-79136d817399") : secret "dns-default-metrics-tls" not found Jul 23 01:28:52.020800 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:52.020580 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls podName:8cd1b24e-6e1e-40fd-84d9-46e49c57c333 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:00.020570389 +0000 UTC m=+52.565519546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls") pod "image-registry-7c4dbbf699-qm7vs" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333") : secret "image-registry-tls" not found Jul 23 01:28:52.428421 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.428389 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5"] Jul 23 01:28:52.462091 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.462064 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5"] Jul 23 01:28:52.462220 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.462159 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:28:52.464784 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.464763 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 23 01:28:52.464914 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.464768 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-q6mqd\"" Jul 23 01:28:52.464914 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.464768 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 23 01:28:52.524170 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.524138 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-xmkj5\" (UID: \"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:28:52.524170 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.524168 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-xmkj5\" (UID: \"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:28:52.625436 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.625397 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-xmkj5\" (UID: \"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:28:52.625604 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.625439 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-xmkj5\" (UID: \"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:28:52.625604 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:52.625552 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 01:28:52.625749 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:52.625651 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert podName:b4514d23-1e8e-4fc5-9f49-ab61bb5c929a nodeName:}" failed. No retries permitted until 2026-07-23 01:28:53.12561206 +0000 UTC m=+45.670561235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-xmkj5" (UID: "b4514d23-1e8e-4fc5-9f49-ab61bb5c929a") : secret "networking-console-plugin-cert" not found Jul 23 01:28:52.626030 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:52.626013 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-xmkj5\" (UID: \"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:28:53.130037 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:53.130003 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-xmkj5\" (UID: \"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:28:53.130513 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:53.130180 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 01:28:53.130513 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:53.130268 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert podName:b4514d23-1e8e-4fc5-9f49-ab61bb5c929a nodeName:}" failed. No retries permitted until 2026-07-23 01:28:54.130247426 +0000 UTC m=+46.675196588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-xmkj5" (UID: "b4514d23-1e8e-4fc5-9f49-ab61bb5c929a") : secret "networking-console-plugin-cert" not found Jul 23 01:28:53.796371 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:53.796335 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tnpqp"] Jul 23 01:28:53.829871 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:53.829834 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tnpqp"] Jul 23 01:28:53.830018 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:53.829952 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tnpqp" Jul 23 01:28:53.832471 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:53.832451 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 23 01:28:53.833387 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:53.833368 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 23 01:28:53.833503 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:53.833486 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zvp9k\"" Jul 23 01:28:53.934727 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:53.934692 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd5b9\" (UniqueName: \"kubernetes.io/projected/1119539a-0bf1-44ac-98b2-1e53fe488cd5-kube-api-access-dd5b9\") pod \"migrator-7cc9c7ffd7-tnpqp\" (UID: \"1119539a-0bf1-44ac-98b2-1e53fe488cd5\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tnpqp" Jul 23 01:28:54.035888 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.035851 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd5b9\" (UniqueName: \"kubernetes.io/projected/1119539a-0bf1-44ac-98b2-1e53fe488cd5-kube-api-access-dd5b9\") pod \"migrator-7cc9c7ffd7-tnpqp\" (UID: \"1119539a-0bf1-44ac-98b2-1e53fe488cd5\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tnpqp" Jul 23 01:28:54.048050 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.047985 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd5b9\" (UniqueName: \"kubernetes.io/projected/1119539a-0bf1-44ac-98b2-1e53fe488cd5-kube-api-access-dd5b9\") pod \"migrator-7cc9c7ffd7-tnpqp\" (UID: \"1119539a-0bf1-44ac-98b2-1e53fe488cd5\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tnpqp" Jul 23 01:28:54.067333 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.057876 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p2kg5"] Jul 23 01:28:54.073420 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.073388 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2kg5"] Jul 23 01:28:54.073570 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.073517 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.078430 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.078403 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 01:28:54.078567 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.078445 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 01:28:54.078567 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.078407 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 01:28:54.078755 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.078736 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 01:28:54.078912 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.078896 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hds2f\"" Jul 23 01:28:54.137183 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.137144 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.137602 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.137212 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21fe7634-3a93-41eb-8801-6ce83ee96fdc-crio-socket\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.137602 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.137237 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.137602 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.137297 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-xmkj5\" (UID: \"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:28:54.137602 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.137328 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpgr6\" (UniqueName: \"kubernetes.io/projected/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-api-access-lpgr6\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.137602 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.137360 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21fe7634-3a93-41eb-8801-6ce83ee96fdc-data-volume\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.137602 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:54.137421 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 01:28:54.137602 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:54.137519 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert podName:b4514d23-1e8e-4fc5-9f49-ab61bb5c929a nodeName:}" failed. No retries permitted until 2026-07-23 01:28:56.137502956 +0000 UTC m=+48.682452107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-xmkj5" (UID: "b4514d23-1e8e-4fc5-9f49-ab61bb5c929a") : secret "networking-console-plugin-cert" not found Jul 23 01:28:54.139140 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.139125 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tnpqp" Jul 23 01:28:54.238236 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.238203 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21fe7634-3a93-41eb-8801-6ce83ee96fdc-crio-socket\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.238236 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.238236 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.238428 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.238354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpgr6\" (UniqueName: \"kubernetes.io/projected/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-api-access-lpgr6\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.238428 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.238376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21fe7634-3a93-41eb-8801-6ce83ee96fdc-data-volume\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.238527 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.238437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.238527 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.238438 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21fe7634-3a93-41eb-8801-6ce83ee96fdc-crio-socket\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.238527 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:54.238435 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 01:28:54.238686 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:54.238546 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm podName:21fe7634-3a93-41eb-8801-6ce83ee96fdc nodeName:}" failed. No retries permitted until 2026-07-23 01:28:54.738532126 +0000 UTC m=+47.283481277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2kg5" (UID: "21fe7634-3a93-41eb-8801-6ce83ee96fdc") : configmap "kube-rbac-proxy" not found Jul 23 01:28:54.238686 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:54.238547 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 01:28:54.238686 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:54.238610 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls podName:21fe7634-3a93-41eb-8801-6ce83ee96fdc nodeName:}" failed. No retries permitted until 2026-07-23 01:28:54.738591404 +0000 UTC m=+47.283540781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2kg5" (UID: "21fe7634-3a93-41eb-8801-6ce83ee96fdc") : secret "insights-runtime-extractor-tls" not found Jul 23 01:28:54.238795 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.238760 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21fe7634-3a93-41eb-8801-6ce83ee96fdc-data-volume\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.247008 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.246984 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpgr6\" (UniqueName: \"kubernetes.io/projected/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-api-access-lpgr6\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.274199 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.274151 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tnpqp"] Jul 23 01:28:54.276609 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:54.276575 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1119539a_0bf1_44ac_98b2_1e53fe488cd5.slice/crio-14210b44dc1d11808c55d3451372d968d905cb6878e2e07654908a31c3b566cf WatchSource:0}: Error finding container 14210b44dc1d11808c55d3451372d968d905cb6878e2e07654908a31c3b566cf: Status 404 returned error can't find the container with id 14210b44dc1d11808c55d3451372d968d905cb6878e2e07654908a31c3b566cf Jul 23 01:28:54.741151 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.741114 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.741349 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:54.741240 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 01:28:54.741349 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:54.741300 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm podName:21fe7634-3a93-41eb-8801-6ce83ee96fdc nodeName:}" failed. No retries permitted until 2026-07-23 01:28:55.741285943 +0000 UTC m=+48.286235094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2kg5" (UID: "21fe7634-3a93-41eb-8801-6ce83ee96fdc") : configmap "kube-rbac-proxy" not found Jul 23 01:28:54.741349 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:54.741247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:54.741349 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:54.741335 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 01:28:54.741526 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:54.741392 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls podName:21fe7634-3a93-41eb-8801-6ce83ee96fdc nodeName:}" failed. No retries permitted until 2026-07-23 01:28:55.741376649 +0000 UTC m=+48.286325801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2kg5" (UID: "21fe7634-3a93-41eb-8801-6ce83ee96fdc") : secret "insights-runtime-extractor-tls" not found Jul 23 01:28:55.227814 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:55.227781 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tnpqp" event={"ID":"1119539a-0bf1-44ac-98b2-1e53fe488cd5","Type":"ContainerStarted","Data":"14210b44dc1d11808c55d3451372d968d905cb6878e2e07654908a31c3b566cf"} Jul 23 01:28:55.748054 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:55.748015 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:55.748217 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:55.748177 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 01:28:55.748262 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:55.748233 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:55.748295 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:55.748270 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm podName:21fe7634-3a93-41eb-8801-6ce83ee96fdc nodeName:}" failed. No retries permitted until 2026-07-23 01:28:57.748249766 +0000 UTC m=+50.293199124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2kg5" (UID: "21fe7634-3a93-41eb-8801-6ce83ee96fdc") : configmap "kube-rbac-proxy" not found Jul 23 01:28:55.748335 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:55.748325 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 01:28:55.748386 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:55.748377 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls podName:21fe7634-3a93-41eb-8801-6ce83ee96fdc nodeName:}" failed. No retries permitted until 2026-07-23 01:28:57.748363728 +0000 UTC m=+50.293312898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2kg5" (UID: "21fe7634-3a93-41eb-8801-6ce83ee96fdc") : secret "insights-runtime-extractor-tls" not found Jul 23 01:28:56.152102 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.152073 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-xmkj5\" (UID: \"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:28:56.152215 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:56.152197 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 01:28:56.152273 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:56.152260 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert podName:b4514d23-1e8e-4fc5-9f49-ab61bb5c929a nodeName:}" failed. No retries permitted until 2026-07-23 01:29:00.152242254 +0000 UTC m=+52.697191405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-xmkj5" (UID: "b4514d23-1e8e-4fc5-9f49-ab61bb5c929a") : secret "networking-console-plugin-cert" not found Jul 23 01:28:56.233584 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.233551 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tnpqp" event={"ID":"1119539a-0bf1-44ac-98b2-1e53fe488cd5","Type":"ContainerStarted","Data":"5e6b6bd52eda8d06099a68ef0303e17aff4b890ec86625513754a11bccf3b9df"} Jul 23 01:28:56.233977 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.233602 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tnpqp" event={"ID":"1119539a-0bf1-44ac-98b2-1e53fe488cd5","Type":"ContainerStarted","Data":"062857b8c473b06a5912ce7390759c7cd87e85d367b1e8d201990031e847f2fe"} Jul 23 01:28:56.249221 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.249174 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-tnpqp" podStartSLOduration=1.572214875 podStartE2EDuration="3.249160257s" totalImagesPullingTime="1.676945382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:28:56.248291382 +0000 UTC m=+48.793240555" watchObservedRunningTime="2026-07-23 01:28:56.249160257 +0000 UTC m=+48.794109429" Jul 23 01:28:56.640331 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.640304 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ssx5d_820cce87-a018-4751-a835-a045e7ccfd8e/dns-node-resolver/0.log" Jul 23 01:28:56.678535 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.678502 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wf8sw"] Jul 23 01:28:56.700261 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.700224 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wf8sw"] Jul 23 01:28:56.700411 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.700319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-wf8sw" Jul 23 01:28:56.702896 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.702867 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 23 01:28:56.703010 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.702917 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-knft4\"" Jul 23 01:28:56.703010 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.702917 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 23 01:28:56.703694 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.703679 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 23 01:28:56.703762 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.703736 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 23 01:28:56.757026 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.756995 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f527846e-6457-482e-97b6-12ba89957efd-signing-cabundle\") pod \"service-ca-58598695f7-wf8sw\" (UID: \"f527846e-6457-482e-97b6-12ba89957efd\") " pod="openshift-service-ca/service-ca-58598695f7-wf8sw" Jul 23 01:28:56.757026 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.757024 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmcxm\" (UniqueName: \"kubernetes.io/projected/f527846e-6457-482e-97b6-12ba89957efd-kube-api-access-vmcxm\") pod \"service-ca-58598695f7-wf8sw\" (UID: \"f527846e-6457-482e-97b6-12ba89957efd\") " pod="openshift-service-ca/service-ca-58598695f7-wf8sw" Jul 23 01:28:56.757184 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.757061 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f527846e-6457-482e-97b6-12ba89957efd-signing-key\") pod \"service-ca-58598695f7-wf8sw\" (UID: \"f527846e-6457-482e-97b6-12ba89957efd\") " pod="openshift-service-ca/service-ca-58598695f7-wf8sw" Jul 23 01:28:56.857702 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.857669 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f527846e-6457-482e-97b6-12ba89957efd-signing-cabundle\") pod \"service-ca-58598695f7-wf8sw\" (UID: \"f527846e-6457-482e-97b6-12ba89957efd\") " pod="openshift-service-ca/service-ca-58598695f7-wf8sw" Jul 23 01:28:56.857702 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.857703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmcxm\" (UniqueName: \"kubernetes.io/projected/f527846e-6457-482e-97b6-12ba89957efd-kube-api-access-vmcxm\") pod \"service-ca-58598695f7-wf8sw\" (UID: \"f527846e-6457-482e-97b6-12ba89957efd\") " pod="openshift-service-ca/service-ca-58598695f7-wf8sw" Jul 23 01:28:56.857850 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.857742 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f527846e-6457-482e-97b6-12ba89957efd-signing-key\") pod \"service-ca-58598695f7-wf8sw\" (UID: \"f527846e-6457-482e-97b6-12ba89957efd\") " pod="openshift-service-ca/service-ca-58598695f7-wf8sw" Jul 23 01:28:56.868740 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.868680 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f527846e-6457-482e-97b6-12ba89957efd-signing-key\") pod \"service-ca-58598695f7-wf8sw\" (UID: \"f527846e-6457-482e-97b6-12ba89957efd\") " pod="openshift-service-ca/service-ca-58598695f7-wf8sw" Jul 23 01:28:56.868927 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.868902 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmcxm\" (UniqueName: \"kubernetes.io/projected/f527846e-6457-482e-97b6-12ba89957efd-kube-api-access-vmcxm\") pod \"service-ca-58598695f7-wf8sw\" (UID: \"f527846e-6457-482e-97b6-12ba89957efd\") " pod="openshift-service-ca/service-ca-58598695f7-wf8sw" Jul 23 01:28:56.868927 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:56.868902 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f527846e-6457-482e-97b6-12ba89957efd-signing-cabundle\") pod \"service-ca-58598695f7-wf8sw\" (UID: \"f527846e-6457-482e-97b6-12ba89957efd\") " pod="openshift-service-ca/service-ca-58598695f7-wf8sw" Jul 23 01:28:57.009450 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:57.009366 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-wf8sw" Jul 23 01:28:57.141684 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:57.141620 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wf8sw"] Jul 23 01:28:57.145239 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:28:57.145211 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf527846e_6457_482e_97b6_12ba89957efd.slice/crio-1794621bad70cc1e267ce07cd3ce2fed8ece60f98f3387b0b379c0703f18d8a7 WatchSource:0}: Error finding container 1794621bad70cc1e267ce07cd3ce2fed8ece60f98f3387b0b379c0703f18d8a7: Status 404 returned error can't find the container with id 1794621bad70cc1e267ce07cd3ce2fed8ece60f98f3387b0b379c0703f18d8a7 Jul 23 01:28:57.236830 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:57.236790 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-wf8sw" event={"ID":"f527846e-6457-482e-97b6-12ba89957efd","Type":"ContainerStarted","Data":"1794621bad70cc1e267ce07cd3ce2fed8ece60f98f3387b0b379c0703f18d8a7"} Jul 23 01:28:57.641417 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:57.641391 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rz68t_50570df0-9213-4039-ab5c-a1347dc88854/node-ca/0.log" Jul 23 01:28:57.765366 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:57.765329 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:57.765537 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:57.765404 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:28:57.765537 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:57.765495 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 01:28:57.765668 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:57.765572 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls podName:21fe7634-3a93-41eb-8801-6ce83ee96fdc nodeName:}" failed. No retries permitted until 2026-07-23 01:29:01.765555079 +0000 UTC m=+54.310504231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2kg5" (UID: "21fe7634-3a93-41eb-8801-6ce83ee96fdc") : secret "insights-runtime-extractor-tls" not found Jul 23 01:28:57.765668 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:57.765499 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 01:28:57.765805 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:28:57.765752 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm podName:21fe7634-3a93-41eb-8801-6ce83ee96fdc nodeName:}" failed. No retries permitted until 2026-07-23 01:29:01.765661082 +0000 UTC m=+54.310610239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2kg5" (UID: "21fe7634-3a93-41eb-8801-6ce83ee96fdc") : configmap "kube-rbac-proxy" not found Jul 23 01:28:58.240398 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:58.240354 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ghjqn_3bef66d3-4454-4b73-b6ce-a9f8b034622f/serve-healthcheck-canary/0.log" Jul 23 01:28:58.443688 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:58.443657 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-tnpqp_1119539a-0bf1-44ac-98b2-1e53fe488cd5/migrator/0.log" Jul 23 01:28:58.641756 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:28:58.641716 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-tnpqp_1119539a-0bf1-44ac-98b2-1e53fe488cd5/graceful-termination/0.log" Jul 23 01:29:00.085693 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:00.085628 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:29:00.086085 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:00.085835 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 01:29:00.086085 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:00.085874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:29:00.086085 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:00.085911 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls podName:75178789-2a57-449a-8fed-79136d817399 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:16.085890223 +0000 UTC m=+68.630839386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls") pod "dns-default-fdh5s" (UID: "75178789-2a57-449a-8fed-79136d817399") : secret "dns-default-metrics-tls" not found Jul 23 01:29:00.086085 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:00.085950 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 01:29:00.086085 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:00.085961 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4dbbf699-qm7vs: secret "image-registry-tls" not found Jul 23 01:29:00.086085 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:00.086000 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls podName:8cd1b24e-6e1e-40fd-84d9-46e49c57c333 nodeName:}" failed. No retries permitted until 2026-07-23 01:29:16.085988189 +0000 UTC m=+68.630937369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls") pod "image-registry-7c4dbbf699-qm7vs" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333") : secret "image-registry-tls" not found Jul 23 01:29:00.187344 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:00.187302 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-xmkj5\" (UID: \"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:29:00.187500 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:00.187428 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 01:29:00.187500 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:00.187487 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert podName:b4514d23-1e8e-4fc5-9f49-ab61bb5c929a nodeName:}" failed. No retries permitted until 2026-07-23 01:29:08.187469863 +0000 UTC m=+60.732419033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-xmkj5" (UID: "b4514d23-1e8e-4fc5-9f49-ab61bb5c929a") : secret "networking-console-plugin-cert" not found Jul 23 01:29:00.245607 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:00.245565 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-wf8sw" event={"ID":"f527846e-6457-482e-97b6-12ba89957efd","Type":"ContainerStarted","Data":"8c726d8e0c181647a41997ec5673f2bffadef0c30af5b2060c274c75919d1a4e"} Jul 23 01:29:00.260854 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:00.260805 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-wf8sw" podStartSLOduration=1.903101102 podStartE2EDuration="4.260791297s" totalImagesPullingTime="2.357690195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:00.260527032 +0000 UTC m=+52.805476207" watchObservedRunningTime="2026-07-23 01:29:00.260791297 +0000 UTC m=+52.805740497" Jul 23 01:29:01.802437 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:01.802396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:29:01.802939 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:01.802456 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:29:01.802939 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:01.802557 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 01:29:01.802939 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:01.802621 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls podName:21fe7634-3a93-41eb-8801-6ce83ee96fdc nodeName:}" failed. No retries permitted until 2026-07-23 01:29:09.802604602 +0000 UTC m=+62.347553752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2kg5" (UID: "21fe7634-3a93-41eb-8801-6ce83ee96fdc") : secret "insights-runtime-extractor-tls" not found Jul 23 01:29:01.803056 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:01.802974 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21fe7634-3a93-41eb-8801-6ce83ee96fdc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:29:08.253997 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:08.253946 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-xmkj5\" (UID: \"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:29:08.254445 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:08.254116 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 01:29:08.254445 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:08.254192 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert podName:b4514d23-1e8e-4fc5-9f49-ab61bb5c929a nodeName:}" failed. No retries permitted until 2026-07-23 01:29:24.254174809 +0000 UTC m=+76.799123959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-xmkj5" (UID: "b4514d23-1e8e-4fc5-9f49-ab61bb5c929a") : secret "networking-console-plugin-cert" not found Jul 23 01:29:09.172756 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:09.172727 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6v4s" Jul 23 01:29:09.868179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:09.868145 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:29:09.870421 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:09.870398 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21fe7634-3a93-41eb-8801-6ce83ee96fdc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kg5\" (UID: \"21fe7634-3a93-41eb-8801-6ce83ee96fdc\") " pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:29:09.986431 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:09.986401 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hds2f\"" Jul 23 01:29:09.994354 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:09.994333 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2kg5" Jul 23 01:29:10.115992 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:10.115944 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2kg5"] Jul 23 01:29:10.118661 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:10.118573 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21fe7634_3a93_41eb_8801_6ce83ee96fdc.slice/crio-8ae7b3f65b07c0832978cc67853bd5f8aed8eb3a2be38ffd832208a74d47b139 WatchSource:0}: Error finding container 8ae7b3f65b07c0832978cc67853bd5f8aed8eb3a2be38ffd832208a74d47b139: Status 404 returned error can't find the container with id 8ae7b3f65b07c0832978cc67853bd5f8aed8eb3a2be38ffd832208a74d47b139 Jul 23 01:29:10.271285 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:10.271245 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2kg5" event={"ID":"21fe7634-3a93-41eb-8801-6ce83ee96fdc","Type":"ContainerStarted","Data":"8ae7b3f65b07c0832978cc67853bd5f8aed8eb3a2be38ffd832208a74d47b139"} Jul 23 01:29:13.697704 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:13.697665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:29:13.700680 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:13.700652 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 01:29:13.710889 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:13.710849 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f94eece0-4626-4e4e-8c9f-d8aab764fe95-original-pull-secret\") pod \"global-pull-secret-syncer-vvpsg\" (UID: \"f94eece0-4626-4e4e-8c9f-d8aab764fe95\") " pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:29:13.779259 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:13.779220 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvpsg" Jul 23 01:29:13.798259 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:13.798230 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:29:13.800951 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:13.800929 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 01:29:13.811146 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:13.811117 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62b44b95-e27a-42a9-b058-0e0363718750-metrics-certs\") pod \"network-metrics-daemon-sgszg\" (UID: \"62b44b95-e27a-42a9-b058-0e0363718750\") " pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:29:13.899138 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:13.899093 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kxq\" (UniqueName: \"kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq\") pod \"network-check-target-dl8vz\" (UID: \"a20bf778-ada3-487a-b3e2-946cbf2366e2\") " pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:29:13.902268 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:13.902244 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 01:29:13.912563 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:13.912531 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 01:29:13.923181 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:13.923150 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8kxq\" (UniqueName: \"kubernetes.io/projected/a20bf778-ada3-487a-b3e2-946cbf2366e2-kube-api-access-m8kxq\") pod \"network-check-target-dl8vz\" (UID: \"a20bf778-ada3-487a-b3e2-946cbf2366e2\") " pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:29:14.088041 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:14.087953 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lg5p2\"" Jul 23 01:29:14.090981 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:14.090956 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-frx6l\"" Jul 23 01:29:14.095672 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:14.095649 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:29:14.099836 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:14.099813 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgszg" Jul 23 01:29:15.780668 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:15.780601 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dl8vz"] Jul 23 01:29:15.784829 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:15.784775 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20bf778_ada3_487a_b3e2_946cbf2366e2.slice/crio-9f4e502c55db815cc255bd125d7fd324c3dda56133c8b1659c702d66f37f4547 WatchSource:0}: Error finding container 9f4e502c55db815cc255bd125d7fd324c3dda56133c8b1659c702d66f37f4547: Status 404 returned error can't find the container with id 9f4e502c55db815cc255bd125d7fd324c3dda56133c8b1659c702d66f37f4547 Jul 23 01:29:15.795440 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:15.795414 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vvpsg"] Jul 23 01:29:15.798330 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:15.798306 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94eece0_4626_4e4e_8c9f_d8aab764fe95.slice/crio-fb9c6055181c84ca8662dce8a28060f397ab5b24ba7420748cea726ab1555d27 WatchSource:0}: Error finding container fb9c6055181c84ca8662dce8a28060f397ab5b24ba7420748cea726ab1555d27: Status 404 returned error can't find the container with id fb9c6055181c84ca8662dce8a28060f397ab5b24ba7420748cea726ab1555d27 Jul 23 01:29:15.815611 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:15.815586 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sgszg"] Jul 23 01:29:15.818174 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:15.818150 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b44b95_e27a_42a9_b058_0e0363718750.slice/crio-fc12abf1f6f03ad4fd415dc6ba4d9c7e04c6214cd2286ad4f0aa0012b25becce WatchSource:0}: Error finding container fc12abf1f6f03ad4fd415dc6ba4d9c7e04c6214cd2286ad4f0aa0012b25becce: Status 404 returned error can't find the container with id fc12abf1f6f03ad4fd415dc6ba4d9c7e04c6214cd2286ad4f0aa0012b25becce Jul 23 01:29:16.116796 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.116760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:29:16.117007 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.116830 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:29:16.119131 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.119106 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75178789-2a57-449a-8fed-79136d817399-metrics-tls\") pod \"dns-default-fdh5s\" (UID: \"75178789-2a57-449a-8fed-79136d817399\") " pod="openshift-dns/dns-default-fdh5s" Jul 23 01:29:16.119234 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.119160 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls\") pod \"image-registry-7c4dbbf699-qm7vs\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:29:16.288822 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.288778 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2kg5" event={"ID":"21fe7634-3a93-41eb-8801-6ce83ee96fdc","Type":"ContainerStarted","Data":"295b0a8b6b924b029b0c272ffb14b3561e140de5b6cdb2a4d1b05187d9bc3560"} Jul 23 01:29:16.290714 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.290676 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vvpsg" event={"ID":"f94eece0-4626-4e4e-8c9f-d8aab764fe95","Type":"ContainerStarted","Data":"fb9c6055181c84ca8662dce8a28060f397ab5b24ba7420748cea726ab1555d27"} Jul 23 01:29:16.291998 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.291953 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sgszg" event={"ID":"62b44b95-e27a-42a9-b058-0e0363718750","Type":"ContainerStarted","Data":"fc12abf1f6f03ad4fd415dc6ba4d9c7e04c6214cd2286ad4f0aa0012b25becce"} Jul 23 01:29:16.293441 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.293407 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dl8vz" event={"ID":"a20bf778-ada3-487a-b3e2-946cbf2366e2","Type":"ContainerStarted","Data":"9f4e502c55db815cc255bd125d7fd324c3dda56133c8b1659c702d66f37f4547"} Jul 23 01:29:16.373998 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.373911 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h6src\"" Jul 23 01:29:16.379901 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.379876 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qjsgq\"" Jul 23 01:29:16.381571 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.381546 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:29:16.388896 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.388868 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fdh5s" Jul 23 01:29:16.563190 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.563159 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c4dbbf699-qm7vs"] Jul 23 01:29:16.566363 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:16.566335 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd1b24e_6e1e_40fd_84d9_46e49c57c333.slice/crio-c28528b85745c8ee87ecb3324d7f7b70192afa6858d4a73148c8cf330ab8b426 WatchSource:0}: Error finding container c28528b85745c8ee87ecb3324d7f7b70192afa6858d4a73148c8cf330ab8b426: Status 404 returned error can't find the container with id c28528b85745c8ee87ecb3324d7f7b70192afa6858d4a73148c8cf330ab8b426 Jul 23 01:29:16.580003 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:16.579972 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fdh5s"] Jul 23 01:29:16.984428 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:16.984389 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75178789_2a57_449a_8fed_79136d817399.slice/crio-ea89d601a0d58596a96d364e105999e8a5065b75e3dc7cbd6df0b280e496635f WatchSource:0}: Error finding container ea89d601a0d58596a96d364e105999e8a5065b75e3dc7cbd6df0b280e496635f: Status 404 returned error can't find the container with id ea89d601a0d58596a96d364e105999e8a5065b75e3dc7cbd6df0b280e496635f Jul 23 01:29:17.302403 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:17.302286 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" event={"ID":"8cd1b24e-6e1e-40fd-84d9-46e49c57c333","Type":"ContainerStarted","Data":"64f2bc79e13da5e6e55f2eacce10b0ad39c5f8712d1bcb2e64ca1a70e9466519"} Jul 23 01:29:17.302403 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:17.302343 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" event={"ID":"8cd1b24e-6e1e-40fd-84d9-46e49c57c333","Type":"ContainerStarted","Data":"c28528b85745c8ee87ecb3324d7f7b70192afa6858d4a73148c8cf330ab8b426"} Jul 23 01:29:17.304021 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:17.303955 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fdh5s" event={"ID":"75178789-2a57-449a-8fed-79136d817399","Type":"ContainerStarted","Data":"ea89d601a0d58596a96d364e105999e8a5065b75e3dc7cbd6df0b280e496635f"} Jul 23 01:29:17.307370 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:17.307193 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:29:17.326348 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:17.326296 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" podStartSLOduration=49.326277025 podStartE2EDuration="49.326277025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:17.324836625 +0000 UTC m=+69.869785798" watchObservedRunningTime="2026-07-23 01:29:17.326277025 +0000 UTC m=+69.871226199" Jul 23 01:29:18.312890 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:18.312771 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sgszg" event={"ID":"62b44b95-e27a-42a9-b058-0e0363718750","Type":"ContainerStarted","Data":"026a8e129aa681609fee0d706b327cb2e9ce348338df2ff74edd4d3e4c7b20d6"} Jul 23 01:29:18.312890 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:18.312846 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sgszg" event={"ID":"62b44b95-e27a-42a9-b058-0e0363718750","Type":"ContainerStarted","Data":"c5d6a82602bbaa4a53e8a84fae7232f4bc3b7d8853650a7840037dd926425a45"} Jul 23 01:29:18.331988 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:18.331556 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sgszg" podStartSLOduration=67.857616499 podStartE2EDuration="1m9.331535929s" totalImagesPullingTime="1.47391943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:18.330476832 +0000 UTC m=+70.875426006" watchObservedRunningTime="2026-07-23 01:29:18.331535929 +0000 UTC m=+70.876485106" Jul 23 01:29:20.753997 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.753879 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2"] Jul 23 01:29:20.761509 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.761473 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv"] Jul 23 01:29:20.761692 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.761607 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2" Jul 23 01:29:20.764413 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.764385 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c4dbbf699-qm7vs"] Jul 23 01:29:20.764556 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.764495 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:20.764623 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.764552 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 23 01:29:20.764623 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.764591 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 01:29:20.764755 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.764684 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-g82gc\"" Jul 23 01:29:20.764875 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.764856 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 01:29:20.764956 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.764860 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 01:29:20.766701 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.766679 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2"] Jul 23 01:29:20.767771 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.767749 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 23 01:29:20.771788 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.771762 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv"] Jul 23 01:29:20.791711 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.791683 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59bf8bfc75-qgnrr"] Jul 23 01:29:20.794919 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.794895 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.807504 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.807479 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59bf8bfc75-qgnrr"] Jul 23 01:29:20.864962 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.864929 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3851da90-548a-4874-9301-392363bdd731-klusterlet-config\") pod \"klusterlet-addon-workmgr-7d465d8645-796qv\" (UID: \"3851da90-548a-4874-9301-392363bdd731\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:20.864962 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.864965 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a444081d-b059-444a-831f-bac40ce1f771-bound-sa-token\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.865183 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.865007 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a444081d-b059-444a-831f-bac40ce1f771-installation-pull-secrets\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.865183 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.865048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a444081d-b059-444a-831f-bac40ce1f771-image-registry-private-configuration\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.865183 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.865079 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42dz\" (UniqueName: \"kubernetes.io/projected/3851da90-548a-4874-9301-392363bdd731-kube-api-access-x42dz\") pod \"klusterlet-addon-workmgr-7d465d8645-796qv\" (UID: \"3851da90-548a-4874-9301-392363bdd731\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:20.865183 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.865103 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5lb\" (UniqueName: \"kubernetes.io/projected/36ef4d4b-ee11-4325-9498-ada16299e0c0-kube-api-access-dk5lb\") pod \"managed-serviceaccount-addon-agent-76b8f46488-drdx2\" (UID: \"36ef4d4b-ee11-4325-9498-ada16299e0c0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2" Jul 23 01:29:20.865183 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.865138 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jrb\" (UniqueName: \"kubernetes.io/projected/a444081d-b059-444a-831f-bac40ce1f771-kube-api-access-j2jrb\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.865183 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.865163 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3851da90-548a-4874-9301-392363bdd731-tmp\") pod \"klusterlet-addon-workmgr-7d465d8645-796qv\" (UID: \"3851da90-548a-4874-9301-392363bdd731\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:20.865463 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.865216 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a444081d-b059-444a-831f-bac40ce1f771-registry-certificates\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.865463 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.865256 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a444081d-b059-444a-831f-bac40ce1f771-registry-tls\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.865463 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.865282 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a444081d-b059-444a-831f-bac40ce1f771-ca-trust-extracted\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.865463 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.865358 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36ef4d4b-ee11-4325-9498-ada16299e0c0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76b8f46488-drdx2\" (UID: \"36ef4d4b-ee11-4325-9498-ada16299e0c0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2" Jul 23 01:29:20.865463 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.865401 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a444081d-b059-444a-831f-bac40ce1f771-trusted-ca\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.965865 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.965828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a444081d-b059-444a-831f-bac40ce1f771-registry-certificates\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.966048 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.965870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a444081d-b059-444a-831f-bac40ce1f771-registry-tls\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.966048 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.965895 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a444081d-b059-444a-831f-bac40ce1f771-ca-trust-extracted\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.966048 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.965930 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36ef4d4b-ee11-4325-9498-ada16299e0c0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76b8f46488-drdx2\" (UID: \"36ef4d4b-ee11-4325-9498-ada16299e0c0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2" Jul 23 01:29:20.966048 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.965959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a444081d-b059-444a-831f-bac40ce1f771-trusted-ca\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.966048 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.965996 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3851da90-548a-4874-9301-392363bdd731-klusterlet-config\") pod \"klusterlet-addon-workmgr-7d465d8645-796qv\" (UID: \"3851da90-548a-4874-9301-392363bdd731\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:20.966048 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.966016 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a444081d-b059-444a-831f-bac40ce1f771-bound-sa-token\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.966341 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.966050 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a444081d-b059-444a-831f-bac40ce1f771-installation-pull-secrets\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.966341 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.966084 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a444081d-b059-444a-831f-bac40ce1f771-image-registry-private-configuration\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.966341 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.966114 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42dz\" (UniqueName: \"kubernetes.io/projected/3851da90-548a-4874-9301-392363bdd731-kube-api-access-x42dz\") pod \"klusterlet-addon-workmgr-7d465d8645-796qv\" (UID: \"3851da90-548a-4874-9301-392363bdd731\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:20.966341 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.966145 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5lb\" (UniqueName: \"kubernetes.io/projected/36ef4d4b-ee11-4325-9498-ada16299e0c0-kube-api-access-dk5lb\") pod \"managed-serviceaccount-addon-agent-76b8f46488-drdx2\" (UID: \"36ef4d4b-ee11-4325-9498-ada16299e0c0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2" Jul 23 01:29:20.966341 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.966189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jrb\" (UniqueName: \"kubernetes.io/projected/a444081d-b059-444a-831f-bac40ce1f771-kube-api-access-j2jrb\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.966341 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.966216 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3851da90-548a-4874-9301-392363bdd731-tmp\") pod \"klusterlet-addon-workmgr-7d465d8645-796qv\" (UID: \"3851da90-548a-4874-9301-392363bdd731\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:20.966627 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.966581 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3851da90-548a-4874-9301-392363bdd731-tmp\") pod \"klusterlet-addon-workmgr-7d465d8645-796qv\" (UID: \"3851da90-548a-4874-9301-392363bdd731\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:20.966871 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.966791 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a444081d-b059-444a-831f-bac40ce1f771-registry-certificates\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.967067 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.967020 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a444081d-b059-444a-831f-bac40ce1f771-ca-trust-extracted\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.967377 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.967344 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a444081d-b059-444a-831f-bac40ce1f771-trusted-ca\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.969208 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.969165 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a444081d-b059-444a-831f-bac40ce1f771-registry-tls\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.969557 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.969513 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36ef4d4b-ee11-4325-9498-ada16299e0c0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76b8f46488-drdx2\" (UID: \"36ef4d4b-ee11-4325-9498-ada16299e0c0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2" Jul 23 01:29:20.969717 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.969696 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3851da90-548a-4874-9301-392363bdd731-klusterlet-config\") pod \"klusterlet-addon-workmgr-7d465d8645-796qv\" (UID: \"3851da90-548a-4874-9301-392363bdd731\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:20.970017 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.969988 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a444081d-b059-444a-831f-bac40ce1f771-installation-pull-secrets\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.970109 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.970038 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a444081d-b059-444a-831f-bac40ce1f771-image-registry-private-configuration\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.975683 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.975618 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jrb\" (UniqueName: \"kubernetes.io/projected/a444081d-b059-444a-831f-bac40ce1f771-kube-api-access-j2jrb\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.975823 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.975783 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a444081d-b059-444a-831f-bac40ce1f771-bound-sa-token\") pod \"image-registry-59bf8bfc75-qgnrr\" (UID: \"a444081d-b059-444a-831f-bac40ce1f771\") " pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:20.975989 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.975962 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5lb\" (UniqueName: \"kubernetes.io/projected/36ef4d4b-ee11-4325-9498-ada16299e0c0-kube-api-access-dk5lb\") pod \"managed-serviceaccount-addon-agent-76b8f46488-drdx2\" (UID: \"36ef4d4b-ee11-4325-9498-ada16299e0c0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2" Jul 23 01:29:20.976093 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:20.976069 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42dz\" (UniqueName: \"kubernetes.io/projected/3851da90-548a-4874-9301-392363bdd731-kube-api-access-x42dz\") pod \"klusterlet-addon-workmgr-7d465d8645-796qv\" (UID: \"3851da90-548a-4874-9301-392363bdd731\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:21.085591 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:21.085510 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2" Jul 23 01:29:21.093948 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:21.093917 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:21.104768 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:21.104702 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:23.520141 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:23.520020 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv"] Jul 23 01:29:23.525050 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:23.524756 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3851da90_548a_4874_9301_392363bdd731.slice/crio-2b257e5f4828377c2ea2449b70f86e8df3f778da69109fc8db07a18a60b4996b WatchSource:0}: Error finding container 2b257e5f4828377c2ea2449b70f86e8df3f778da69109fc8db07a18a60b4996b: Status 404 returned error can't find the container with id 2b257e5f4828377c2ea2449b70f86e8df3f778da69109fc8db07a18a60b4996b Jul 23 01:29:23.549997 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:23.549811 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2"] Jul 23 01:29:23.553951 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:23.553899 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59bf8bfc75-qgnrr"] Jul 23 01:29:23.555088 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:23.555065 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36ef4d4b_ee11_4325_9498_ada16299e0c0.slice/crio-1faaf09505b6e1f068977f913359b9152627d88597f9563a5b1eaa6f7a3e4539 WatchSource:0}: Error finding container 1faaf09505b6e1f068977f913359b9152627d88597f9563a5b1eaa6f7a3e4539: Status 404 returned error can't find the container with id 1faaf09505b6e1f068977f913359b9152627d88597f9563a5b1eaa6f7a3e4539 Jul 23 01:29:23.558258 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:23.558235 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda444081d_b059_444a_831f_bac40ce1f771.slice/crio-72454e35e457ff19870b14806f93126b086dba984e7152a34028b8315cda2e33 WatchSource:0}: Error finding container 72454e35e457ff19870b14806f93126b086dba984e7152a34028b8315cda2e33: Status 404 returned error can't find the container with id 72454e35e457ff19870b14806f93126b086dba984e7152a34028b8315cda2e33 Jul 23 01:29:24.297912 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.297817 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-xmkj5\" (UID: \"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:29:24.301403 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.301316 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4514d23-1e8e-4fc5-9f49-ab61bb5c929a-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-xmkj5\" (UID: \"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:29:24.341060 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.341020 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" event={"ID":"3851da90-548a-4874-9301-392363bdd731","Type":"ContainerStarted","Data":"2b257e5f4828377c2ea2449b70f86e8df3f778da69109fc8db07a18a60b4996b"} Jul 23 01:29:24.344290 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.343960 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dl8vz" event={"ID":"a20bf778-ada3-487a-b3e2-946cbf2366e2","Type":"ContainerStarted","Data":"c0cd46b1b63f67798a96bbcad5dd6d95520eaf515882250b5ec6ee02c25f3f46"} Jul 23 01:29:24.344290 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.344039 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:29:24.347240 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.347181 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2" event={"ID":"36ef4d4b-ee11-4325-9498-ada16299e0c0","Type":"ContainerStarted","Data":"1faaf09505b6e1f068977f913359b9152627d88597f9563a5b1eaa6f7a3e4539"} Jul 23 01:29:24.349411 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.349385 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2kg5" event={"ID":"21fe7634-3a93-41eb-8801-6ce83ee96fdc","Type":"ContainerStarted","Data":"6233bb2b9d9f710f249380635f127036175c9490e79cbb13e0d37dbbd7190e93"} Jul 23 01:29:24.351384 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.351358 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vvpsg" event={"ID":"f94eece0-4626-4e4e-8c9f-d8aab764fe95","Type":"ContainerStarted","Data":"0998007000eba61531b47260badef6e1fd1f2fc0eda9e8a3cc74a21deae07d25"} Jul 23 01:29:24.353534 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.353508 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fdh5s" event={"ID":"75178789-2a57-449a-8fed-79136d817399","Type":"ContainerStarted","Data":"cd88cbfd6c4c3eaa4c2208008e9255a5fe4e45da8d34a79678bcb0a9ab9b17e6"} Jul 23 01:29:24.353663 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.353556 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fdh5s" event={"ID":"75178789-2a57-449a-8fed-79136d817399","Type":"ContainerStarted","Data":"d6f2539ba9a7ad8dedf1b64918ed5a6204d5ea017b966cf44a320368a4df882c"} Jul 23 01:29:24.354018 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.353997 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fdh5s" Jul 23 01:29:24.355582 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.355558 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" event={"ID":"a444081d-b059-444a-831f-bac40ce1f771","Type":"ContainerStarted","Data":"e5f1e9a0eecfee88f4d2b22855bae025bbb0d7873c49df4b75ec41ae6b01af5f"} Jul 23 01:29:24.355702 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.355600 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" event={"ID":"a444081d-b059-444a-831f-bac40ce1f771","Type":"ContainerStarted","Data":"72454e35e457ff19870b14806f93126b086dba984e7152a34028b8315cda2e33"} Jul 23 01:29:24.356046 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.356027 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:24.360346 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.360270 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dl8vz" podStartSLOduration=67.814313517 podStartE2EDuration="1m15.360254603s" totalImagesPullingTime="7.545941086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:24.359453544 +0000 UTC m=+76.904402720" watchObservedRunningTime="2026-07-23 01:29:24.360254603 +0000 UTC m=+76.905203779" Jul 23 01:29:24.374742 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.374396 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vvpsg" podStartSLOduration=67.840906379 podStartE2EDuration="1m15.374379568s" totalImagesPullingTime="7.533473189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:24.373061119 +0000 UTC m=+76.918010316" watchObservedRunningTime="2026-07-23 01:29:24.374379568 +0000 UTC m=+76.919328722" Jul 23 01:29:24.393229 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.393155 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" podStartSLOduration=4.393140899 podStartE2EDuration="4.393140899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:24.391565421 +0000 UTC m=+76.936514614" watchObservedRunningTime="2026-07-23 01:29:24.393140899 +0000 UTC m=+76.938090052" Jul 23 01:29:24.412055 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.410944 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fdh5s" podStartSLOduration=34.060644946 podStartE2EDuration="40.410925385s" totalImagesPullingTime="6.350280439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:24.410001137 +0000 UTC m=+76.954950335" watchObservedRunningTime="2026-07-23 01:29:24.410925385 +0000 UTC m=+76.955874559" Jul 23 01:29:24.480749 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.480715 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4"] Jul 23 01:29:24.484223 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.483955 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4" Jul 23 01:29:24.490371 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.489521 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 23 01:29:24.490371 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.489801 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7z6g6\"" Jul 23 01:29:24.490924 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.490825 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4"] Jul 23 01:29:24.575238 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.574458 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-q6mqd\"" Jul 23 01:29:24.582537 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.582512 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" Jul 23 01:29:24.600042 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.599931 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/55f9819e-440c-45f9-bcd9-72ef8ed5e14b-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6rxf4\" (UID: \"55f9819e-440c-45f9-bcd9-72ef8ed5e14b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4" Jul 23 01:29:24.703702 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.703128 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/55f9819e-440c-45f9-bcd9-72ef8ed5e14b-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6rxf4\" (UID: \"55f9819e-440c-45f9-bcd9-72ef8ed5e14b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4" Jul 23 01:29:24.703702 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:24.703325 2572 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 23 01:29:24.703702 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:24.703396 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f9819e-440c-45f9-bcd9-72ef8ed5e14b-tls-certificates podName:55f9819e-440c-45f9-bcd9-72ef8ed5e14b nodeName:}" failed. No retries permitted until 2026-07-23 01:29:25.203373106 +0000 UTC m=+77.748322263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/55f9819e-440c-45f9-bcd9-72ef8ed5e14b-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-6rxf4" (UID: "55f9819e-440c-45f9-bcd9-72ef8ed5e14b") : secret "prometheus-operator-admission-webhook-tls" not found Jul 23 01:29:24.784539 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:24.784424 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5"] Jul 23 01:29:24.790165 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:24.790117 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4514d23_1e8e_4fc5_9f49_ab61bb5c929a.slice/crio-f5094554214aaf87b257498a1e04833e91061abe9d1802481f870b9dba242999 WatchSource:0}: Error finding container f5094554214aaf87b257498a1e04833e91061abe9d1802481f870b9dba242999: Status 404 returned error can't find the container with id f5094554214aaf87b257498a1e04833e91061abe9d1802481f870b9dba242999 Jul 23 01:29:25.160148 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.159009 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-rcvbl"] Jul 23 01:29:25.163897 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.163852 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-rcvbl" Jul 23 01:29:25.167093 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.167067 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 01:29:25.167350 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.167332 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wcmlt\"" Jul 23 01:29:25.167868 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.167693 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 01:29:25.176226 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.176139 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-rcvbl"] Jul 23 01:29:25.209663 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.209596 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/55f9819e-440c-45f9-bcd9-72ef8ed5e14b-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6rxf4\" (UID: \"55f9819e-440c-45f9-bcd9-72ef8ed5e14b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4" Jul 23 01:29:25.218749 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.217371 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/55f9819e-440c-45f9-bcd9-72ef8ed5e14b-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6rxf4\" (UID: \"55f9819e-440c-45f9-bcd9-72ef8ed5e14b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4" Jul 23 01:29:25.310413 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.310372 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvjn5\" (UniqueName: \"kubernetes.io/projected/eaabba99-6e4f-4c8e-9ff3-f49edbafcc2a-kube-api-access-jvjn5\") pod \"downloads-5bc67dcd46-rcvbl\" (UID: \"eaabba99-6e4f-4c8e-9ff3-f49edbafcc2a\") " pod="openshift-console/downloads-5bc67dcd46-rcvbl" Jul 23 01:29:25.363993 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.363910 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" event={"ID":"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a","Type":"ContainerStarted","Data":"f5094554214aaf87b257498a1e04833e91061abe9d1802481f870b9dba242999"} Jul 23 01:29:25.409625 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.409588 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4" Jul 23 01:29:25.411342 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.410877 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvjn5\" (UniqueName: \"kubernetes.io/projected/eaabba99-6e4f-4c8e-9ff3-f49edbafcc2a-kube-api-access-jvjn5\") pod \"downloads-5bc67dcd46-rcvbl\" (UID: \"eaabba99-6e4f-4c8e-9ff3-f49edbafcc2a\") " pod="openshift-console/downloads-5bc67dcd46-rcvbl" Jul 23 01:29:25.420396 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.420368 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvjn5\" (UniqueName: \"kubernetes.io/projected/eaabba99-6e4f-4c8e-9ff3-f49edbafcc2a-kube-api-access-jvjn5\") pod \"downloads-5bc67dcd46-rcvbl\" (UID: \"eaabba99-6e4f-4c8e-9ff3-f49edbafcc2a\") " pod="openshift-console/downloads-5bc67dcd46-rcvbl" Jul 23 01:29:25.477916 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:25.477880 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-rcvbl" Jul 23 01:29:29.905573 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:29.905499 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4"] Jul 23 01:29:29.908354 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:29.908314 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f9819e_440c_45f9_bcd9_72ef8ed5e14b.slice/crio-622917d660cff8a5b057a6bc6eb9a1d0945a392f5987a12f8888338cff203343 WatchSource:0}: Error finding container 622917d660cff8a5b057a6bc6eb9a1d0945a392f5987a12f8888338cff203343: Status 404 returned error can't find the container with id 622917d660cff8a5b057a6bc6eb9a1d0945a392f5987a12f8888338cff203343 Jul 23 01:29:29.919524 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:29.918904 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-rcvbl"] Jul 23 01:29:29.921674 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:29.921617 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaabba99_6e4f_4c8e_9ff3_f49edbafcc2a.slice/crio-f28496279b0fd0c2292101b3a591f47eb55b2f3dc87d3db6166166a4dcd979bc WatchSource:0}: Error finding container f28496279b0fd0c2292101b3a591f47eb55b2f3dc87d3db6166166a4dcd979bc: Status 404 returned error can't find the container with id f28496279b0fd0c2292101b3a591f47eb55b2f3dc87d3db6166166a4dcd979bc Jul 23 01:29:30.383241 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:30.383196 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-rcvbl" event={"ID":"eaabba99-6e4f-4c8e-9ff3-f49edbafcc2a","Type":"ContainerStarted","Data":"f28496279b0fd0c2292101b3a591f47eb55b2f3dc87d3db6166166a4dcd979bc"} Jul 23 01:29:30.385319 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:30.385285 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2kg5" event={"ID":"21fe7634-3a93-41eb-8801-6ce83ee96fdc","Type":"ContainerStarted","Data":"7f673871369ff95730a997a60c65f3a141db7dc4e5b0c44f7e7e77ea4647018d"} Jul 23 01:29:30.386675 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:30.386654 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" event={"ID":"3851da90-548a-4874-9301-392363bdd731","Type":"ContainerStarted","Data":"701fb1e6bdcfedb8c81bac010ed21b4ad0c6857458aba514b823172880164d66"} Jul 23 01:29:30.386819 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:30.386798 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:30.388211 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:30.388188 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" event={"ID":"b4514d23-1e8e-4fc5-9f49-ab61bb5c929a","Type":"ContainerStarted","Data":"6525f3f3527ad823e9c6d59196c6ab3ccc8c280e8da14761773bd3ebf9f03666"} Jul 23 01:29:30.388429 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:30.388408 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" Jul 23 01:29:30.389518 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:30.389499 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2" event={"ID":"36ef4d4b-ee11-4325-9498-ada16299e0c0","Type":"ContainerStarted","Data":"217e167df4bf7cb350ec7ae71c707b1891ecb34332532c4cfb9100dc5c338888"} Jul 23 01:29:30.390480 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:30.390459 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4" event={"ID":"55f9819e-440c-45f9-bcd9-72ef8ed5e14b","Type":"ContainerStarted","Data":"622917d660cff8a5b057a6bc6eb9a1d0945a392f5987a12f8888338cff203343"} Jul 23 01:29:30.402692 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:30.402655 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p2kg5" podStartSLOduration=17.02226726 podStartE2EDuration="36.402622877s" totalImagesPullingTime="19.380355617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:28:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:30.402021675 +0000 UTC m=+82.946970851" watchObservedRunningTime="2026-07-23 01:29:30.402622877 +0000 UTC m=+82.947572069" Jul 23 01:29:30.418568 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:30.418530 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76b8f46488-drdx2" podStartSLOduration=4.222934309 podStartE2EDuration="10.418518698s" totalImagesPullingTime="6.195584389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:30.417922312 +0000 UTC m=+82.962871516" watchObservedRunningTime="2026-07-23 01:29:30.418518698 +0000 UTC m=+82.963467848" Jul 23 01:29:30.431696 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:30.431629 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-xmkj5" podStartSLOduration=33.47238152 podStartE2EDuration="38.431617587s" totalImagesPullingTime="4.959236067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:30.431540518 +0000 UTC m=+82.976489692" watchObservedRunningTime="2026-07-23 01:29:30.431617587 +0000 UTC m=+82.976566762" Jul 23 01:29:30.446874 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:30.446815 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d465d8645-796qv" podStartSLOduration=4.204504693 podStartE2EDuration="10.446801476s" totalImagesPullingTime="6.242296783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:30.44593986 +0000 UTC m=+82.990889052" watchObservedRunningTime="2026-07-23 01:29:30.446801476 +0000 UTC m=+82.991750652" Jul 23 01:29:31.395023 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:31.394930 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4" event={"ID":"55f9819e-440c-45f9-bcd9-72ef8ed5e14b","Type":"ContainerStarted","Data":"a1ad4d00e07c93a73b5378b48693838f6fa3fac5c4a74b373b575342a508f5e1"} Jul 23 01:29:31.395497 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:31.395153 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4" Jul 23 01:29:31.399787 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:31.399761 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4" Jul 23 01:29:31.410796 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:31.410753 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6rxf4" podStartSLOduration=6.188477415 podStartE2EDuration="7.410739186s" totalImagesPullingTime="1.222261771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:31.41004395 +0000 UTC m=+83.954993144" watchObservedRunningTime="2026-07-23 01:29:31.410739186 +0000 UTC m=+83.955688360" Jul 23 01:29:34.366761 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:34.366726 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fdh5s" Jul 23 01:29:36.936325 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:36.936291 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kjxjh"] Jul 23 01:29:36.940921 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:36.940900 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:36.944325 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:36.944299 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 01:29:36.944458 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:36.944299 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 01:29:36.944693 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:36.944673 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nh4h7\"" Jul 23 01:29:36.945016 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:36.944996 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 01:29:36.945094 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:36.945045 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 01:29:36.945136 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:36.944996 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 01:29:36.945304 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:36.945287 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 01:29:37.106128 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.106083 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-accelerators-collector-config\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.106307 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.106161 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cad8448-e3b6-4e82-bb12-520a9e2adeec-metrics-client-ca\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.106307 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.106188 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.106307 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.106217 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glzdk\" (UniqueName: \"kubernetes.io/projected/7cad8448-e3b6-4e82-bb12-520a9e2adeec-kube-api-access-glzdk\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.106307 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.106266 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-textfile\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.106307 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.106295 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cad8448-e3b6-4e82-bb12-520a9e2adeec-root\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.106559 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.106330 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-tls\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.106559 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.106384 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cad8448-e3b6-4e82-bb12-520a9e2adeec-sys\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.106559 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.106429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-wtmp\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.208012 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.207471 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cad8448-e3b6-4e82-bb12-520a9e2adeec-metrics-client-ca\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.208012 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.207514 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.208012 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.207563 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glzdk\" (UniqueName: \"kubernetes.io/projected/7cad8448-e3b6-4e82-bb12-520a9e2adeec-kube-api-access-glzdk\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.208012 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.207613 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-textfile\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.208012 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.207655 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cad8448-e3b6-4e82-bb12-520a9e2adeec-root\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.208012 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.207691 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-tls\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.208012 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.207724 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cad8448-e3b6-4e82-bb12-520a9e2adeec-sys\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.208012 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.207752 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-wtmp\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.208012 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.207791 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-accelerators-collector-config\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.208553 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.208285 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cad8448-e3b6-4e82-bb12-520a9e2adeec-metrics-client-ca\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.208616 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.208592 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cad8448-e3b6-4e82-bb12-520a9e2adeec-sys\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.209154 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.208715 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cad8448-e3b6-4e82-bb12-520a9e2adeec-root\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.209154 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.208785 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-wtmp\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.209154 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:37.208806 2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 01:29:37.209154 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.208810 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-textfile\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.209154 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:29:37.208861 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-tls podName:7cad8448-e3b6-4e82-bb12-520a9e2adeec nodeName:}" failed. No retries permitted until 2026-07-23 01:29:37.708843405 +0000 UTC m=+90.253792561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-tls") pod "node-exporter-kjxjh" (UID: "7cad8448-e3b6-4e82-bb12-520a9e2adeec") : secret "node-exporter-tls" not found Jul 23 01:29:37.209154 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.209106 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-accelerators-collector-config\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.211687 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.211139 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.218480 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.218438 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glzdk\" (UniqueName: \"kubernetes.io/projected/7cad8448-e3b6-4e82-bb12-520a9e2adeec-kube-api-access-glzdk\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.711997 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.711960 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-tls\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.714768 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.714741 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cad8448-e3b6-4e82-bb12-520a9e2adeec-node-exporter-tls\") pod \"node-exporter-kjxjh\" (UID: \"7cad8448-e3b6-4e82-bb12-520a9e2adeec\") " pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.853909 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:37.853729 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kjxjh" Jul 23 01:29:37.866959 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:37.866920 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cad8448_e3b6_4e82_bb12_520a9e2adeec.slice/crio-58e780377938c85b47aaa8bbf2b9ccb148385641c3a89bbb49096d95234e038b WatchSource:0}: Error finding container 58e780377938c85b47aaa8bbf2b9ccb148385641c3a89bbb49096d95234e038b: Status 404 returned error can't find the container with id 58e780377938c85b47aaa8bbf2b9ccb148385641c3a89bbb49096d95234e038b Jul 23 01:29:38.425230 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:38.425101 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kjxjh" event={"ID":"7cad8448-e3b6-4e82-bb12-520a9e2adeec","Type":"ContainerStarted","Data":"58e780377938c85b47aaa8bbf2b9ccb148385641c3a89bbb49096d95234e038b"} Jul 23 01:29:39.431347 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:39.431313 2572 generic.go:356] "Generic (PLEG): container finished" podID="7cad8448-e3b6-4e82-bb12-520a9e2adeec" containerID="cff46e864b1dc09184861f9b735f9b98c051fb7c19b53b6dd00625bdd875fdef" exitCode=0 Jul 23 01:29:39.431833 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:39.431370 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kjxjh" event={"ID":"7cad8448-e3b6-4e82-bb12-520a9e2adeec","Type":"ContainerDied","Data":"cff46e864b1dc09184861f9b735f9b98c051fb7c19b53b6dd00625bdd875fdef"} Jul 23 01:29:40.438221 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:40.438177 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kjxjh" event={"ID":"7cad8448-e3b6-4e82-bb12-520a9e2adeec","Type":"ContainerStarted","Data":"c48b919fd31e5060f453490e63bdec3ce9a9cbfef2d977fdb25d65274845902b"} Jul 23 01:29:40.438711 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:40.438253 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kjxjh" event={"ID":"7cad8448-e3b6-4e82-bb12-520a9e2adeec","Type":"ContainerStarted","Data":"d890dff06d1c793d8e23b0f7af4e456b041c25b40e55d2646f188967ebbd9a05"} Jul 23 01:29:40.461036 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:40.460979 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kjxjh" podStartSLOduration=3.340206837 podStartE2EDuration="4.460963149s" totalImagesPullingTime="1.048520101s" totalInitContainerRuntime="72.236211ms" isStatefulPod=true podCreationTimestamp="2026-07-23 01:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:40.458656868 +0000 UTC m=+93.003606037" watchObservedRunningTime="2026-07-23 01:29:40.460963149 +0000 UTC m=+93.005912300" Jul 23 01:29:40.771551 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:40.771476 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:29:41.111077 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:41.111022 2572 patch_prober.go:28] interesting pod/image-registry-59bf8bfc75-qgnrr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 01:29:41.111238 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:41.111088 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" podUID="a444081d-b059-444a-831f-bac40ce1f771" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 01:29:45.791707 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:45.791660 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" podUID="8cd1b24e-6e1e-40fd-84d9-46e49c57c333" containerName="registry" containerID="cri-o://64f2bc79e13da5e6e55f2eacce10b0ad39c5f8712d1bcb2e64ca1a70e9466519" gracePeriod=30 Jul 23 01:29:46.373676 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.373630 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59bf8bfc75-qgnrr" Jul 23 01:29:46.701530 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.701441 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54db56f99c-pkvv7"] Jul 23 01:29:46.764444 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.764366 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54db56f99c-pkvv7"] Jul 23 01:29:46.764651 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.764454 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.767097 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.767068 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 01:29:46.767228 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.767126 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z9qsp\"" Jul 23 01:29:46.767447 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.767373 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 01:29:46.767447 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.767377 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 01:29:46.767967 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.767939 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 01:29:46.768078 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.767985 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 01:29:46.776624 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.776600 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 01:29:46.776899 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.776873 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-config\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.777009 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.776949 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-oauth-serving-cert\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.777009 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.776998 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-oauth-config\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.777121 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.777038 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-service-ca\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.777121 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.777088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-serving-cert\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.777196 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.777120 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dmhq\" (UniqueName: \"kubernetes.io/projected/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-kube-api-access-7dmhq\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.777196 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.777169 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-trusted-ca-bundle\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.877934 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.877897 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-trusted-ca-bundle\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.878413 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.877953 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-config\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.878413 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.878004 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-oauth-serving-cert\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.878413 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.878036 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-oauth-config\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.878413 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.878075 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-service-ca\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.878413 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.878106 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-serving-cert\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.878703 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.878455 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dmhq\" (UniqueName: \"kubernetes.io/projected/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-kube-api-access-7dmhq\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.878880 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.878846 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-config\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.879054 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.879035 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-trusted-ca-bundle\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.879378 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.879356 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-oauth-serving-cert\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.879546 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.879523 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-service-ca\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.880891 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.880859 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-serving-cert\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.880891 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.880866 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-oauth-config\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:46.887715 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:46.887620 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dmhq\" (UniqueName: \"kubernetes.io/projected/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-kube-api-access-7dmhq\") pod \"console-54db56f99c-pkvv7\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:47.077968 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:47.077930 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:48.469855 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.469825 2572 generic.go:356] "Generic (PLEG): container finished" podID="8cd1b24e-6e1e-40fd-84d9-46e49c57c333" containerID="64f2bc79e13da5e6e55f2eacce10b0ad39c5f8712d1bcb2e64ca1a70e9466519" exitCode=0 Jul 23 01:29:48.470222 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.469893 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" event={"ID":"8cd1b24e-6e1e-40fd-84d9-46e49c57c333","Type":"ContainerDied","Data":"64f2bc79e13da5e6e55f2eacce10b0ad39c5f8712d1bcb2e64ca1a70e9466519"} Jul 23 01:29:48.609715 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.609691 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:29:48.638391 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.638359 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54db56f99c-pkvv7"] Jul 23 01:29:48.641095 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:29:48.641069 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2de6f7d_3da2_4bf8_b492_eb6aa6470f0d.slice/crio-baefd0296aad9f96f7e2158d8ac9e1fa7a221997d0ab8a977670c77b923fe101 WatchSource:0}: Error finding container baefd0296aad9f96f7e2158d8ac9e1fa7a221997d0ab8a977670c77b923fe101: Status 404 returned error can't find the container with id baefd0296aad9f96f7e2158d8ac9e1fa7a221997d0ab8a977670c77b923fe101 Jul 23 01:29:48.693760 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.693737 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls\") pod \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " Jul 23 01:29:48.693883 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.693788 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-ca-trust-extracted\") pod \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " Jul 23 01:29:48.693883 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.693812 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-bound-sa-token\") pod \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " Jul 23 01:29:48.693883 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.693829 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62rjg\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-kube-api-access-62rjg\") pod \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " Jul 23 01:29:48.693883 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.693846 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-certificates\") pod \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " Jul 23 01:29:48.694097 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.693894 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-image-registry-private-configuration\") pod \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " Jul 23 01:29:48.694097 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.693925 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-trusted-ca\") pod \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " Jul 23 01:29:48.694097 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.693964 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-installation-pull-secrets\") pod \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\" (UID: \"8cd1b24e-6e1e-40fd-84d9-46e49c57c333\") " Jul 23 01:29:48.694401 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.694363 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8cd1b24e-6e1e-40fd-84d9-46e49c57c333" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:29:48.694918 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.694888 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8cd1b24e-6e1e-40fd-84d9-46e49c57c333" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:29:48.696144 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.696119 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8cd1b24e-6e1e-40fd-84d9-46e49c57c333" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:29:48.696230 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.696137 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8cd1b24e-6e1e-40fd-84d9-46e49c57c333" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:29:48.696337 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.696316 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8cd1b24e-6e1e-40fd-84d9-46e49c57c333" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:29:48.696415 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.696396 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-kube-api-access-62rjg" (OuterVolumeSpecName: "kube-api-access-62rjg") pod "8cd1b24e-6e1e-40fd-84d9-46e49c57c333" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333"). InnerVolumeSpecName "kube-api-access-62rjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:29:48.696452 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.696423 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8cd1b24e-6e1e-40fd-84d9-46e49c57c333" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:29:48.702573 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.702553 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8cd1b24e-6e1e-40fd-84d9-46e49c57c333" (UID: "8cd1b24e-6e1e-40fd-84d9-46e49c57c333"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:29:48.795431 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.795393 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-tls\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:29:48.795431 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.795431 2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-ca-trust-extracted\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:29:48.795624 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.795447 2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-bound-sa-token\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:29:48.795624 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.795461 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62rjg\" (UniqueName: \"kubernetes.io/projected/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-kube-api-access-62rjg\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:29:48.795624 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.795474 2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-registry-certificates\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:29:48.795624 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.795489 2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-image-registry-private-configuration\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:29:48.795624 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.795503 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-trusted-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:29:48.795624 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:48.795516 2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cd1b24e-6e1e-40fd-84d9-46e49c57c333-installation-pull-secrets\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:29:49.477779 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:49.477735 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-rcvbl" event={"ID":"eaabba99-6e4f-4c8e-9ff3-f49edbafcc2a","Type":"ContainerStarted","Data":"8fef6585aa8b097055b0269e8a0fe7e737d6c502f9180bdf8b731baebe9b2b0b"} Jul 23 01:29:49.478629 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:49.478351 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-rcvbl" Jul 23 01:29:49.480398 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:49.480356 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" event={"ID":"8cd1b24e-6e1e-40fd-84d9-46e49c57c333","Type":"ContainerDied","Data":"c28528b85745c8ee87ecb3324d7f7b70192afa6858d4a73148c8cf330ab8b426"} Jul 23 01:29:49.480521 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:49.480381 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4dbbf699-qm7vs" Jul 23 01:29:49.480521 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:49.480423 2572 scope.go:117] "RemoveContainer" containerID="64f2bc79e13da5e6e55f2eacce10b0ad39c5f8712d1bcb2e64ca1a70e9466519" Jul 23 01:29:49.482160 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:49.481858 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54db56f99c-pkvv7" event={"ID":"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d","Type":"ContainerStarted","Data":"baefd0296aad9f96f7e2158d8ac9e1fa7a221997d0ab8a977670c77b923fe101"} Jul 23 01:29:49.492194 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:49.491792 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-rcvbl" Jul 23 01:29:49.502036 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:49.501985 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-rcvbl" podStartSLOduration=5.7997394589999995 podStartE2EDuration="24.501967465s" totalImagesPullingTime="18.702228006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:49.501250474 +0000 UTC m=+102.046199648" watchObservedRunningTime="2026-07-23 01:29:49.501967465 +0000 UTC m=+102.046916638" Jul 23 01:29:49.539501 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:49.539460 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c4dbbf699-qm7vs"] Jul 23 01:29:49.545854 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:49.545814 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c4dbbf699-qm7vs"] Jul 23 01:29:49.977466 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:49.977427 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cd1b24e-6e1e-40fd-84d9-46e49c57c333" path="/var/lib/kubelet/pods/8cd1b24e-6e1e-40fd-84d9-46e49c57c333/volumes" Jul 23 01:29:52.501583 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:52.501489 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54db56f99c-pkvv7" event={"ID":"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d","Type":"ContainerStarted","Data":"11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd"} Jul 23 01:29:52.522867 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:52.522798 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54db56f99c-pkvv7" podStartSLOduration=2.817381251 podStartE2EDuration="6.522775561s" totalImagesPullingTime="3.70539431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:29:52.520981579 +0000 UTC m=+105.065930756" watchObservedRunningTime="2026-07-23 01:29:52.522775561 +0000 UTC m=+105.067724735" Jul 23 01:29:55.367409 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:55.367375 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dl8vz" Jul 23 01:29:57.078905 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:57.078869 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:57.079385 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:57.078954 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:57.084601 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:57.084578 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:29:57.525759 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:29:57.525727 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:30:55.405932 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.405896 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78b744475f-j7ml4"] Jul 23 01:30:55.406376 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.406163 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd1b24e-6e1e-40fd-84d9-46e49c57c333" containerName="registry" Jul 23 01:30:55.406376 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.406174 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd1b24e-6e1e-40fd-84d9-46e49c57c333" containerName="registry" Jul 23 01:30:55.406376 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.406213 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd1b24e-6e1e-40fd-84d9-46e49c57c333" containerName="registry" Jul 23 01:30:55.408715 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.408698 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.421574 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.421542 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-config\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.421713 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.421650 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-trusted-ca-bundle\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.421713 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.421704 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-service-ca\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.421816 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.421734 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgckv\" (UniqueName: \"kubernetes.io/projected/0adba6f5-c7f0-45e9-9655-690c1e450c1f-kube-api-access-tgckv\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.421816 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.421756 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-oauth-serving-cert\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.421895 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.421812 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-oauth-config\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.421895 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.421878 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-serving-cert\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.422733 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.422705 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78b744475f-j7ml4"] Jul 23 01:30:55.522251 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.522215 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-serving-cert\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.522441 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.522279 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-config\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.522441 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.522306 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-trusted-ca-bundle\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.522441 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.522325 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-service-ca\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.522441 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.522345 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgckv\" (UniqueName: \"kubernetes.io/projected/0adba6f5-c7f0-45e9-9655-690c1e450c1f-kube-api-access-tgckv\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.522441 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.522367 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-oauth-serving-cert\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.522441 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.522389 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-oauth-config\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.523194 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.523157 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-oauth-serving-cert\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.523307 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.523239 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-config\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.523367 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.523322 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-trusted-ca-bundle\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.523406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.523378 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-service-ca\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.524821 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.524800 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-serving-cert\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.524937 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.524915 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-oauth-config\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.532001 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.531980 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgckv\" (UniqueName: \"kubernetes.io/projected/0adba6f5-c7f0-45e9-9655-690c1e450c1f-kube-api-access-tgckv\") pod \"console-78b744475f-j7ml4\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.721889 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.721801 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:30:55.846927 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:55.846879 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78b744475f-j7ml4"] Jul 23 01:30:55.849709 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:30:55.849675 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adba6f5_c7f0_45e9_9655_690c1e450c1f.slice/crio-bd4dde65702b1fc8f9eb8d9cf9d32dbd0240d6b31ab8c3249673e326c66b3f6e WatchSource:0}: Error finding container bd4dde65702b1fc8f9eb8d9cf9d32dbd0240d6b31ab8c3249673e326c66b3f6e: Status 404 returned error can't find the container with id bd4dde65702b1fc8f9eb8d9cf9d32dbd0240d6b31ab8c3249673e326c66b3f6e Jul 23 01:30:56.727511 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:56.727470 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78b744475f-j7ml4" event={"ID":"0adba6f5-c7f0-45e9-9655-690c1e450c1f","Type":"ContainerStarted","Data":"152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a"} Jul 23 01:30:56.727928 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:56.727530 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78b744475f-j7ml4" event={"ID":"0adba6f5-c7f0-45e9-9655-690c1e450c1f","Type":"ContainerStarted","Data":"bd4dde65702b1fc8f9eb8d9cf9d32dbd0240d6b31ab8c3249673e326c66b3f6e"} Jul 23 01:30:56.747550 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:30:56.747470 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78b744475f-j7ml4" podStartSLOduration=1.747455963 podStartE2EDuration="1.747455963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:30:56.746392153 +0000 UTC m=+169.291341326" watchObservedRunningTime="2026-07-23 01:30:56.747455963 +0000 UTC m=+169.292405135" Jul 23 01:31:05.722624 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:05.722525 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:31:05.722624 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:05.722617 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:31:05.727343 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:05.727317 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:31:05.763716 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:05.763689 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:31:05.822457 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:05.822424 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54db56f99c-pkvv7"] Jul 23 01:31:30.842473 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:30.842414 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54db56f99c-pkvv7" podUID="d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" containerName="console" containerID="cri-o://11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd" gracePeriod=15 Jul 23 01:31:31.081799 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.081772 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:31:31.190592 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.190553 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-oauth-serving-cert\") pod \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " Jul 23 01:31:31.190592 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.190595 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-trusted-ca-bundle\") pod \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " Jul 23 01:31:31.190839 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.190672 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-oauth-config\") pod \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " Jul 23 01:31:31.190839 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.190703 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-config\") pod \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " Jul 23 01:31:31.190839 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.190718 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-service-ca\") pod \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " Jul 23 01:31:31.190839 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.190742 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-serving-cert\") pod \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " Jul 23 01:31:31.190839 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.190757 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dmhq\" (UniqueName: \"kubernetes.io/projected/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-kube-api-access-7dmhq\") pod \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\" (UID: \"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d\") " Jul 23 01:31:31.191167 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.191135 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" (UID: "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:31:31.191272 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.191180 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-service-ca" (OuterVolumeSpecName: "service-ca") pod "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" (UID: "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:31:31.191272 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.191208 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-config" (OuterVolumeSpecName: "console-config") pod "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" (UID: "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:31:31.191349 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.191294 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" (UID: "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:31:31.193030 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.193000 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" (UID: "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:31:31.193161 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.193048 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-kube-api-access-7dmhq" (OuterVolumeSpecName: "kube-api-access-7dmhq") pod "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" (UID: "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d"). InnerVolumeSpecName "kube-api-access-7dmhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:31:31.193161 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.193101 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" (UID: "d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:31:31.291610 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.291562 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-oauth-config\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:31:31.291610 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.291602 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-config\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:31:31.291610 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.291615 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:31:31.291878 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.291625 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-console-serving-cert\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:31:31.291878 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.291663 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dmhq\" (UniqueName: \"kubernetes.io/projected/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-kube-api-access-7dmhq\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:31:31.291878 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.291676 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-oauth-serving-cert\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:31:31.291878 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.291687 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d-trusted-ca-bundle\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:31:31.850485 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.850448 2572 generic.go:356] "Generic (PLEG): container finished" podID="d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" containerID="11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd" exitCode=0 Jul 23 01:31:31.850944 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.850500 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54db56f99c-pkvv7" Jul 23 01:31:31.850944 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.850540 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54db56f99c-pkvv7" event={"ID":"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d","Type":"ContainerDied","Data":"11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd"} Jul 23 01:31:31.850944 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.850594 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54db56f99c-pkvv7" event={"ID":"d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d","Type":"ContainerDied","Data":"baefd0296aad9f96f7e2158d8ac9e1fa7a221997d0ab8a977670c77b923fe101"} Jul 23 01:31:31.850944 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.850615 2572 scope.go:117] "RemoveContainer" containerID="11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd" Jul 23 01:31:31.864320 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.864299 2572 scope.go:117] "RemoveContainer" containerID="11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd" Jul 23 01:31:31.864601 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:31:31.864578 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd\": container with ID starting with 11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd not found: ID does not exist" containerID="11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd" Jul 23 01:31:31.864665 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.864609 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd"} err="failed to get container status \"11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd\": rpc error: code = NotFound desc = could not find container \"11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd\": container with ID starting with 11afb7e500de5d2f4931b6fceca4bfac0a4254947604e3180869fe3b1c0324cd not found: ID does not exist" Jul 23 01:31:31.873273 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.873251 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54db56f99c-pkvv7"] Jul 23 01:31:31.876935 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.876914 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54db56f99c-pkvv7"] Jul 23 01:31:31.973907 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:31:31.973872 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" path="/var/lib/kubelet/pods/d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d/volumes" Jul 23 01:32:14.978543 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:14.978506 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f85c8bbb6-xckr4"] Jul 23 01:32:14.978963 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:14.978771 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" containerName="console" Jul 23 01:32:14.978963 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:14.978782 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" containerName="console" Jul 23 01:32:14.978963 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:14.978841 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2de6f7d-3da2-4bf8-b492-eb6aa6470f0d" containerName="console" Jul 23 01:32:14.981193 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:14.981173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:14.992375 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:14.992351 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f85c8bbb6-xckr4"] Jul 23 01:32:15.000165 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.000139 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da94f987-03b0-45ec-822a-88c01b5071e4-console-serving-cert\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.000289 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.000179 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-trusted-ca-bundle\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.000289 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.000225 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-console-config\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.000392 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.000294 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5js7c\" (UniqueName: \"kubernetes.io/projected/da94f987-03b0-45ec-822a-88c01b5071e4-kube-api-access-5js7c\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.000392 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.000377 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-oauth-serving-cert\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.000473 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.000421 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da94f987-03b0-45ec-822a-88c01b5071e4-console-oauth-config\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.000473 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.000447 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-service-ca\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.101701 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.101664 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-oauth-serving-cert\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.101870 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.101710 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da94f987-03b0-45ec-822a-88c01b5071e4-console-oauth-config\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.101870 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.101732 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-service-ca\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.101870 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.101760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da94f987-03b0-45ec-822a-88c01b5071e4-console-serving-cert\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.102003 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.101922 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-trusted-ca-bundle\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.102003 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.101954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-console-config\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.102003 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.101988 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5js7c\" (UniqueName: \"kubernetes.io/projected/da94f987-03b0-45ec-822a-88c01b5071e4-kube-api-access-5js7c\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.102524 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.102498 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-oauth-serving-cert\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.102657 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.102570 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-service-ca\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.102706 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.102629 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-console-config\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.103193 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.103166 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-trusted-ca-bundle\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.104194 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.104176 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da94f987-03b0-45ec-822a-88c01b5071e4-console-serving-cert\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.104298 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.104244 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da94f987-03b0-45ec-822a-88c01b5071e4-console-oauth-config\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.109822 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.109800 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5js7c\" (UniqueName: \"kubernetes.io/projected/da94f987-03b0-45ec-822a-88c01b5071e4-kube-api-access-5js7c\") pod \"console-7f85c8bbb6-xckr4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.290328 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.290228 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:15.412657 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:15.412608 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f85c8bbb6-xckr4"] Jul 23 01:32:15.414652 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:32:15.414605 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda94f987_03b0_45ec_822a_88c01b5071e4.slice/crio-3d6231aa41e3029b071a7abd02c0eb23c76e8aed87a89d5f3a9b5dba37ba6b86 WatchSource:0}: Error finding container 3d6231aa41e3029b071a7abd02c0eb23c76e8aed87a89d5f3a9b5dba37ba6b86: Status 404 returned error can't find the container with id 3d6231aa41e3029b071a7abd02c0eb23c76e8aed87a89d5f3a9b5dba37ba6b86 Jul 23 01:32:16.006604 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:16.006560 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f85c8bbb6-xckr4" event={"ID":"da94f987-03b0-45ec-822a-88c01b5071e4","Type":"ContainerStarted","Data":"bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608"} Jul 23 01:32:16.007115 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:16.006620 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f85c8bbb6-xckr4" event={"ID":"da94f987-03b0-45ec-822a-88c01b5071e4","Type":"ContainerStarted","Data":"3d6231aa41e3029b071a7abd02c0eb23c76e8aed87a89d5f3a9b5dba37ba6b86"} Jul 23 01:32:16.023332 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:16.023284 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f85c8bbb6-xckr4" podStartSLOduration=2.023270644 podStartE2EDuration="2.023270644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:32:16.022770255 +0000 UTC m=+248.567719447" watchObservedRunningTime="2026-07-23 01:32:16.023270644 +0000 UTC m=+248.568219816" Jul 23 01:32:25.291187 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:25.291131 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:25.291187 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:25.291205 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:25.295771 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:25.295751 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:26.048668 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:26.048623 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:32:26.093610 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:26.093580 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78b744475f-j7ml4"] Jul 23 01:32:51.113596 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.113538 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78b744475f-j7ml4" podUID="0adba6f5-c7f0-45e9-9655-690c1e450c1f" containerName="console" containerID="cri-o://152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a" gracePeriod=15 Jul 23 01:32:51.366165 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.366099 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:32:51.479335 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.479304 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-trusted-ca-bundle\") pod \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " Jul 23 01:32:51.479335 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.479336 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-service-ca\") pod \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " Jul 23 01:32:51.479590 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.479359 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-serving-cert\") pod \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " Jul 23 01:32:51.479590 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.479392 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-oauth-config\") pod \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " Jul 23 01:32:51.479590 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.479413 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-oauth-serving-cert\") pod \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " Jul 23 01:32:51.479590 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.479436 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-config\") pod \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " Jul 23 01:32:51.479590 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.479454 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgckv\" (UniqueName: \"kubernetes.io/projected/0adba6f5-c7f0-45e9-9655-690c1e450c1f-kube-api-access-tgckv\") pod \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\" (UID: \"0adba6f5-c7f0-45e9-9655-690c1e450c1f\") " Jul 23 01:32:51.479864 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.479763 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0adba6f5-c7f0-45e9-9655-690c1e450c1f" (UID: "0adba6f5-c7f0-45e9-9655-690c1e450c1f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:32:51.480017 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.479990 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-service-ca" (OuterVolumeSpecName: "service-ca") pod "0adba6f5-c7f0-45e9-9655-690c1e450c1f" (UID: "0adba6f5-c7f0-45e9-9655-690c1e450c1f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:32:51.480102 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.480037 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0adba6f5-c7f0-45e9-9655-690c1e450c1f" (UID: "0adba6f5-c7f0-45e9-9655-690c1e450c1f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:32:51.480158 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.480126 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-config" (OuterVolumeSpecName: "console-config") pod "0adba6f5-c7f0-45e9-9655-690c1e450c1f" (UID: "0adba6f5-c7f0-45e9-9655-690c1e450c1f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:32:51.481610 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.481590 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0adba6f5-c7f0-45e9-9655-690c1e450c1f" (UID: "0adba6f5-c7f0-45e9-9655-690c1e450c1f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:32:51.481610 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.481585 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adba6f5-c7f0-45e9-9655-690c1e450c1f-kube-api-access-tgckv" (OuterVolumeSpecName: "kube-api-access-tgckv") pod "0adba6f5-c7f0-45e9-9655-690c1e450c1f" (UID: "0adba6f5-c7f0-45e9-9655-690c1e450c1f"). InnerVolumeSpecName "kube-api-access-tgckv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:32:51.481762 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.481610 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0adba6f5-c7f0-45e9-9655-690c1e450c1f" (UID: "0adba6f5-c7f0-45e9-9655-690c1e450c1f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:32:51.580112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.580079 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-config\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:32:51.580112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.580107 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgckv\" (UniqueName: \"kubernetes.io/projected/0adba6f5-c7f0-45e9-9655-690c1e450c1f-kube-api-access-tgckv\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:32:51.580112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.580117 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-trusted-ca-bundle\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:32:51.580328 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.580126 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:32:51.580328 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.580134 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-serving-cert\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:32:51.580328 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.580142 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0adba6f5-c7f0-45e9-9655-690c1e450c1f-console-oauth-config\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:32:51.580328 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:51.580150 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0adba6f5-c7f0-45e9-9655-690c1e450c1f-oauth-serving-cert\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:32:52.131450 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:52.131417 2572 generic.go:356] "Generic (PLEG): container finished" podID="0adba6f5-c7f0-45e9-9655-690c1e450c1f" containerID="152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a" exitCode=0 Jul 23 01:32:52.131913 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:52.131464 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78b744475f-j7ml4" Jul 23 01:32:52.131913 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:52.131486 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78b744475f-j7ml4" event={"ID":"0adba6f5-c7f0-45e9-9655-690c1e450c1f","Type":"ContainerDied","Data":"152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a"} Jul 23 01:32:52.131913 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:52.131526 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78b744475f-j7ml4" event={"ID":"0adba6f5-c7f0-45e9-9655-690c1e450c1f","Type":"ContainerDied","Data":"bd4dde65702b1fc8f9eb8d9cf9d32dbd0240d6b31ab8c3249673e326c66b3f6e"} Jul 23 01:32:52.131913 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:52.131546 2572 scope.go:117] "RemoveContainer" containerID="152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a" Jul 23 01:32:52.139305 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:52.139288 2572 scope.go:117] "RemoveContainer" containerID="152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a" Jul 23 01:32:52.139555 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:32:52.139536 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a\": container with ID starting with 152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a not found: ID does not exist" containerID="152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a" Jul 23 01:32:52.139628 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:52.139565 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a"} err="failed to get container status \"152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a\": rpc error: code = NotFound desc = could not find container \"152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a\": container with ID starting with 152a37dd9132572cb3f40e51a026eefc1e019a87a902b3defbd00130f46a518a not found: ID does not exist" Jul 23 01:32:52.146652 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:52.146614 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78b744475f-j7ml4"] Jul 23 01:32:52.150327 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:52.150307 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78b744475f-j7ml4"] Jul 23 01:32:53.974789 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:32:53.974755 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0adba6f5-c7f0-45e9-9655-690c1e450c1f" path="/var/lib/kubelet/pods/0adba6f5-c7f0-45e9-9655-690c1e450c1f/volumes" Jul 23 01:33:07.871026 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:07.870995 2572 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 01:33:13.951347 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:13.951307 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq"] Jul 23 01:33:13.952739 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:13.951599 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0adba6f5-c7f0-45e9-9655-690c1e450c1f" containerName="console" Jul 23 01:33:13.952739 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:13.951610 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0adba6f5-c7f0-45e9-9655-690c1e450c1f" containerName="console" Jul 23 01:33:13.952739 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:13.951669 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0adba6f5-c7f0-45e9-9655-690c1e450c1f" containerName="console" Jul 23 01:33:13.953395 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:13.953378 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:13.957702 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:13.957680 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tqqgm\"" Jul 23 01:33:13.957835 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:13.957683 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 01:33:13.957835 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:13.957683 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 01:33:13.965644 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:13.965611 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq"] Jul 23 01:33:14.049018 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:14.048986 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f95adb5-7478-4d4e-a38f-879476e626bd-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq\" (UID: \"5f95adb5-7478-4d4e-a38f-879476e626bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:14.049203 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:14.049033 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f95adb5-7478-4d4e-a38f-879476e626bd-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq\" (UID: \"5f95adb5-7478-4d4e-a38f-879476e626bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:14.049203 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:14.049137 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6kv7\" (UniqueName: \"kubernetes.io/projected/5f95adb5-7478-4d4e-a38f-879476e626bd-kube-api-access-v6kv7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq\" (UID: \"5f95adb5-7478-4d4e-a38f-879476e626bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:14.149890 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:14.149851 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6kv7\" (UniqueName: \"kubernetes.io/projected/5f95adb5-7478-4d4e-a38f-879476e626bd-kube-api-access-v6kv7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq\" (UID: \"5f95adb5-7478-4d4e-a38f-879476e626bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:14.149890 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:14.149901 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f95adb5-7478-4d4e-a38f-879476e626bd-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq\" (UID: \"5f95adb5-7478-4d4e-a38f-879476e626bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:14.150108 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:14.149923 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f95adb5-7478-4d4e-a38f-879476e626bd-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq\" (UID: \"5f95adb5-7478-4d4e-a38f-879476e626bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:14.150262 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:14.150247 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f95adb5-7478-4d4e-a38f-879476e626bd-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq\" (UID: \"5f95adb5-7478-4d4e-a38f-879476e626bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:14.150344 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:14.150319 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f95adb5-7478-4d4e-a38f-879476e626bd-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq\" (UID: \"5f95adb5-7478-4d4e-a38f-879476e626bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:14.160573 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:14.160538 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6kv7\" (UniqueName: \"kubernetes.io/projected/5f95adb5-7478-4d4e-a38f-879476e626bd-kube-api-access-v6kv7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq\" (UID: \"5f95adb5-7478-4d4e-a38f-879476e626bd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:14.262092 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:14.261996 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:14.385196 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:14.385086 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq"] Jul 23 01:33:14.390430 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:33:14.389747 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f95adb5_7478_4d4e_a38f_879476e626bd.slice/crio-8677f31cb43f1453539aae7beaae40fce1984927e9d575266b98f438be29559b WatchSource:0}: Error finding container 8677f31cb43f1453539aae7beaae40fce1984927e9d575266b98f438be29559b: Status 404 returned error can't find the container with id 8677f31cb43f1453539aae7beaae40fce1984927e9d575266b98f438be29559b Jul 23 01:33:14.391608 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:14.391585 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 01:33:15.210257 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:15.210218 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" event={"ID":"5f95adb5-7478-4d4e-a38f-879476e626bd","Type":"ContainerStarted","Data":"8677f31cb43f1453539aae7beaae40fce1984927e9d575266b98f438be29559b"} Jul 23 01:33:21.235567 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:21.235524 2572 generic.go:356] "Generic (PLEG): container finished" podID="5f95adb5-7478-4d4e-a38f-879476e626bd" containerID="5ff4d8f77c3756cd044b08af1996c1f03f740aacc721a35686df5b7f2a270373" exitCode=0 Jul 23 01:33:21.235976 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:21.235581 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" event={"ID":"5f95adb5-7478-4d4e-a38f-879476e626bd","Type":"ContainerDied","Data":"5ff4d8f77c3756cd044b08af1996c1f03f740aacc721a35686df5b7f2a270373"} Jul 23 01:33:23.245040 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:23.245004 2572 generic.go:356] "Generic (PLEG): container finished" podID="5f95adb5-7478-4d4e-a38f-879476e626bd" containerID="f04aac97eaf4be17d20b768f66453f2f5f8f7f670350ce6cef0e584f154063d3" exitCode=0 Jul 23 01:33:23.245427 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:23.245066 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" event={"ID":"5f95adb5-7478-4d4e-a38f-879476e626bd","Type":"ContainerDied","Data":"f04aac97eaf4be17d20b768f66453f2f5f8f7f670350ce6cef0e584f154063d3"} Jul 23 01:33:30.273454 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:30.273415 2572 generic.go:356] "Generic (PLEG): container finished" podID="5f95adb5-7478-4d4e-a38f-879476e626bd" containerID="1bbd4f959f38df465bdfacc00ad64ddf63a646eb15cfe1e5d36b87233468e4a5" exitCode=0 Jul 23 01:33:30.273894 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:30.273475 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" event={"ID":"5f95adb5-7478-4d4e-a38f-879476e626bd","Type":"ContainerDied","Data":"1bbd4f959f38df465bdfacc00ad64ddf63a646eb15cfe1e5d36b87233468e4a5"} Jul 23 01:33:31.397496 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:31.397468 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:31.590489 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:31.590396 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f95adb5-7478-4d4e-a38f-879476e626bd-util\") pod \"5f95adb5-7478-4d4e-a38f-879476e626bd\" (UID: \"5f95adb5-7478-4d4e-a38f-879476e626bd\") " Jul 23 01:33:31.590489 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:31.590440 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6kv7\" (UniqueName: \"kubernetes.io/projected/5f95adb5-7478-4d4e-a38f-879476e626bd-kube-api-access-v6kv7\") pod \"5f95adb5-7478-4d4e-a38f-879476e626bd\" (UID: \"5f95adb5-7478-4d4e-a38f-879476e626bd\") " Jul 23 01:33:31.590489 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:31.590470 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f95adb5-7478-4d4e-a38f-879476e626bd-bundle\") pod \"5f95adb5-7478-4d4e-a38f-879476e626bd\" (UID: \"5f95adb5-7478-4d4e-a38f-879476e626bd\") " Jul 23 01:33:31.591187 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:31.591161 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f95adb5-7478-4d4e-a38f-879476e626bd-bundle" (OuterVolumeSpecName: "bundle") pod "5f95adb5-7478-4d4e-a38f-879476e626bd" (UID: "5f95adb5-7478-4d4e-a38f-879476e626bd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:33:31.592602 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:31.592568 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f95adb5-7478-4d4e-a38f-879476e626bd-kube-api-access-v6kv7" (OuterVolumeSpecName: "kube-api-access-v6kv7") pod "5f95adb5-7478-4d4e-a38f-879476e626bd" (UID: "5f95adb5-7478-4d4e-a38f-879476e626bd"). InnerVolumeSpecName "kube-api-access-v6kv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:33:31.597320 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:31.597291 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f95adb5-7478-4d4e-a38f-879476e626bd-util" (OuterVolumeSpecName: "util") pod "5f95adb5-7478-4d4e-a38f-879476e626bd" (UID: "5f95adb5-7478-4d4e-a38f-879476e626bd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:33:31.691323 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:31.691289 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f95adb5-7478-4d4e-a38f-879476e626bd-util\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:33:31.691323 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:31.691316 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6kv7\" (UniqueName: \"kubernetes.io/projected/5f95adb5-7478-4d4e-a38f-879476e626bd-kube-api-access-v6kv7\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:33:31.691323 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:31.691327 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f95adb5-7478-4d4e-a38f-879476e626bd-bundle\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:33:32.282315 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:32.282231 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" event={"ID":"5f95adb5-7478-4d4e-a38f-879476e626bd","Type":"ContainerDied","Data":"8677f31cb43f1453539aae7beaae40fce1984927e9d575266b98f438be29559b"} Jul 23 01:33:32.282315 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:32.282267 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cvgwq" Jul 23 01:33:32.282471 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:32.282274 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8677f31cb43f1453539aae7beaae40fce1984927e9d575266b98f438be29559b" Jul 23 01:33:36.700993 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.700951 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl"] Jul 23 01:33:36.701457 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.701319 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f95adb5-7478-4d4e-a38f-879476e626bd" containerName="util" Jul 23 01:33:36.701457 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.701338 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f95adb5-7478-4d4e-a38f-879476e626bd" containerName="util" Jul 23 01:33:36.701457 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.701349 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f95adb5-7478-4d4e-a38f-879476e626bd" containerName="extract" Jul 23 01:33:36.701457 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.701358 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f95adb5-7478-4d4e-a38f-879476e626bd" containerName="extract" Jul 23 01:33:36.701457 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.701368 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f95adb5-7478-4d4e-a38f-879476e626bd" containerName="pull" Jul 23 01:33:36.701457 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.701375 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f95adb5-7478-4d4e-a38f-879476e626bd" containerName="pull" Jul 23 01:33:36.701457 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.701433 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f95adb5-7478-4d4e-a38f-879476e626bd" containerName="extract" Jul 23 01:33:36.735315 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.735278 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl"] Jul 23 01:33:36.735480 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.735388 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl" Jul 23 01:33:36.737875 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.737852 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 23 01:33:36.737985 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.737891 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 23 01:33:36.738149 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.738130 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ns9zb\"" Jul 23 01:33:36.829240 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.829191 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gth6n\" (UniqueName: \"kubernetes.io/projected/d63e2ca8-86da-4518-a9f1-6add8aa2f4f1-kube-api-access-gth6n\") pod \"cert-manager-operator-controller-manager-55c444bf86-j59bl\" (UID: \"d63e2ca8-86da-4518-a9f1-6add8aa2f4f1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl" Jul 23 01:33:36.829417 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.829268 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d63e2ca8-86da-4518-a9f1-6add8aa2f4f1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j59bl\" (UID: \"d63e2ca8-86da-4518-a9f1-6add8aa2f4f1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl" Jul 23 01:33:36.929684 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.929624 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gth6n\" (UniqueName: \"kubernetes.io/projected/d63e2ca8-86da-4518-a9f1-6add8aa2f4f1-kube-api-access-gth6n\") pod \"cert-manager-operator-controller-manager-55c444bf86-j59bl\" (UID: \"d63e2ca8-86da-4518-a9f1-6add8aa2f4f1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl" Jul 23 01:33:36.929843 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.929703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d63e2ca8-86da-4518-a9f1-6add8aa2f4f1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j59bl\" (UID: \"d63e2ca8-86da-4518-a9f1-6add8aa2f4f1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl" Jul 23 01:33:36.930043 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.930026 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d63e2ca8-86da-4518-a9f1-6add8aa2f4f1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j59bl\" (UID: \"d63e2ca8-86da-4518-a9f1-6add8aa2f4f1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl" Jul 23 01:33:36.937879 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:36.937855 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gth6n\" (UniqueName: \"kubernetes.io/projected/d63e2ca8-86da-4518-a9f1-6add8aa2f4f1-kube-api-access-gth6n\") pod \"cert-manager-operator-controller-manager-55c444bf86-j59bl\" (UID: \"d63e2ca8-86da-4518-a9f1-6add8aa2f4f1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl" Jul 23 01:33:37.044132 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:37.044040 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl" Jul 23 01:33:37.197594 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:37.197560 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl"] Jul 23 01:33:37.201350 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:33:37.201325 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63e2ca8_86da_4518_a9f1_6add8aa2f4f1.slice/crio-028b6c7c4f42b9e4e814248e1f864eacad9ab0026b48ae20c2318e603dac0bf1 WatchSource:0}: Error finding container 028b6c7c4f42b9e4e814248e1f864eacad9ab0026b48ae20c2318e603dac0bf1: Status 404 returned error can't find the container with id 028b6c7c4f42b9e4e814248e1f864eacad9ab0026b48ae20c2318e603dac0bf1 Jul 23 01:33:37.301409 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:37.301329 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl" event={"ID":"d63e2ca8-86da-4518-a9f1-6add8aa2f4f1","Type":"ContainerStarted","Data":"028b6c7c4f42b9e4e814248e1f864eacad9ab0026b48ae20c2318e603dac0bf1"} Jul 23 01:33:39.311430 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:39.311340 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl" event={"ID":"d63e2ca8-86da-4518-a9f1-6add8aa2f4f1","Type":"ContainerStarted","Data":"549ecc5355dbd4453514fffb055940bec5d6eeaa06c2120c54573a6a126d981f"} Jul 23 01:33:39.331368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:39.331313 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j59bl" podStartSLOduration=1.5817233480000001 podStartE2EDuration="3.331294376s" totalImagesPullingTime="1.749571028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:33:39.330191613 +0000 UTC m=+331.875140804" watchObservedRunningTime="2026-07-23 01:33:39.331294376 +0000 UTC m=+331.876243531" Jul 23 01:33:40.547685 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.547648 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc"] Jul 23 01:33:40.549949 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.549933 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:40.552161 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.552137 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tqqgm\"" Jul 23 01:33:40.552300 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.552193 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 01:33:40.552984 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.552968 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 01:33:40.561110 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.561087 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc"] Jul 23 01:33:40.563793 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.563772 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8v27\" (UniqueName: \"kubernetes.io/projected/c7b05400-d663-4df8-84e3-4689ddc165f0-kube-api-access-c8v27\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc\" (UID: \"c7b05400-d663-4df8-84e3-4689ddc165f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:40.563864 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.563807 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7b05400-d663-4df8-84e3-4689ddc165f0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc\" (UID: \"c7b05400-d663-4df8-84e3-4689ddc165f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:40.563968 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.563938 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7b05400-d663-4df8-84e3-4689ddc165f0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc\" (UID: \"c7b05400-d663-4df8-84e3-4689ddc165f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:40.665049 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.665018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7b05400-d663-4df8-84e3-4689ddc165f0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc\" (UID: \"c7b05400-d663-4df8-84e3-4689ddc165f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:40.665250 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.665079 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8v27\" (UniqueName: \"kubernetes.io/projected/c7b05400-d663-4df8-84e3-4689ddc165f0-kube-api-access-c8v27\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc\" (UID: \"c7b05400-d663-4df8-84e3-4689ddc165f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:40.665250 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.665105 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7b05400-d663-4df8-84e3-4689ddc165f0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc\" (UID: \"c7b05400-d663-4df8-84e3-4689ddc165f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:40.665465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.665440 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7b05400-d663-4df8-84e3-4689ddc165f0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc\" (UID: \"c7b05400-d663-4df8-84e3-4689ddc165f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:40.665529 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.665471 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7b05400-d663-4df8-84e3-4689ddc165f0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc\" (UID: \"c7b05400-d663-4df8-84e3-4689ddc165f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:40.674573 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.674543 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8v27\" (UniqueName: \"kubernetes.io/projected/c7b05400-d663-4df8-84e3-4689ddc165f0-kube-api-access-c8v27\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc\" (UID: \"c7b05400-d663-4df8-84e3-4689ddc165f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:40.858253 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.858214 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:40.982025 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:40.981932 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc"] Jul 23 01:33:40.984505 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:33:40.984471 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b05400_d663_4df8_84e3_4689ddc165f0.slice/crio-0d3b13e7ebd980f2694fc9c81b5bc859c141e3003b9202a2b1326ed8570371a1 WatchSource:0}: Error finding container 0d3b13e7ebd980f2694fc9c81b5bc859c141e3003b9202a2b1326ed8570371a1: Status 404 returned error can't find the container with id 0d3b13e7ebd980f2694fc9c81b5bc859c141e3003b9202a2b1326ed8570371a1 Jul 23 01:33:41.322321 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:41.322234 2572 generic.go:356] "Generic (PLEG): container finished" podID="c7b05400-d663-4df8-84e3-4689ddc165f0" containerID="63d4350704b81fc001fec2535c5c6898b346a08ca6a997d1d8aa9ee84215ff23" exitCode=0 Jul 23 01:33:41.322478 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:41.322327 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" event={"ID":"c7b05400-d663-4df8-84e3-4689ddc165f0","Type":"ContainerDied","Data":"63d4350704b81fc001fec2535c5c6898b346a08ca6a997d1d8aa9ee84215ff23"} Jul 23 01:33:41.322478 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:41.322389 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" event={"ID":"c7b05400-d663-4df8-84e3-4689ddc165f0","Type":"ContainerStarted","Data":"0d3b13e7ebd980f2694fc9c81b5bc859c141e3003b9202a2b1326ed8570371a1"} Jul 23 01:33:43.334288 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.334254 2572 generic.go:356] "Generic (PLEG): container finished" podID="c7b05400-d663-4df8-84e3-4689ddc165f0" containerID="78e93e7f703f7db29194ab2e9e3e2ecc032f6025fce9ddeaf325624ae90a363c" exitCode=0 Jul 23 01:33:43.334669 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.334340 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" event={"ID":"c7b05400-d663-4df8-84e3-4689ddc165f0","Type":"ContainerDied","Data":"78e93e7f703f7db29194ab2e9e3e2ecc032f6025fce9ddeaf325624ae90a363c"} Jul 23 01:33:43.380790 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.380764 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fclxs"] Jul 23 01:33:43.383685 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.383661 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fclxs" Jul 23 01:33:43.387684 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.387662 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 01:33:43.387684 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.387675 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 01:33:43.387826 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.387665 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xpwtt\"" Jul 23 01:33:43.394972 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.394951 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fclxs"] Jul 23 01:33:43.489047 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.489015 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4qt7\" (UniqueName: \"kubernetes.io/projected/7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5-kube-api-access-l4qt7\") pod \"cert-manager-cainjector-559cdd8c87-fclxs\" (UID: \"7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fclxs" Jul 23 01:33:43.489179 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.489074 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fclxs\" (UID: \"7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fclxs" Jul 23 01:33:43.590298 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.590262 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fclxs\" (UID: \"7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fclxs" Jul 23 01:33:43.590476 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.590350 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4qt7\" (UniqueName: \"kubernetes.io/projected/7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5-kube-api-access-l4qt7\") pod \"cert-manager-cainjector-559cdd8c87-fclxs\" (UID: \"7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fclxs" Jul 23 01:33:43.598616 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.598582 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fclxs\" (UID: \"7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fclxs" Jul 23 01:33:43.599021 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.599001 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4qt7\" (UniqueName: \"kubernetes.io/projected/7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5-kube-api-access-l4qt7\") pod \"cert-manager-cainjector-559cdd8c87-fclxs\" (UID: \"7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fclxs" Jul 23 01:33:43.692186 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.692154 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fclxs" Jul 23 01:33:43.816707 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:43.816681 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fclxs"] Jul 23 01:33:43.818716 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:33:43.818687 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b4b55eb_77c5_41c2_9ccc_c9e785b5c7a5.slice/crio-eff7b28cf4c400f4e9c496eb3b487f3cd82ce2bfaccf7ff247d4139d80f70156 WatchSource:0}: Error finding container eff7b28cf4c400f4e9c496eb3b487f3cd82ce2bfaccf7ff247d4139d80f70156: Status 404 returned error can't find the container with id eff7b28cf4c400f4e9c496eb3b487f3cd82ce2bfaccf7ff247d4139d80f70156 Jul 23 01:33:44.343907 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:44.343867 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fclxs" event={"ID":"7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5","Type":"ContainerStarted","Data":"eff7b28cf4c400f4e9c496eb3b487f3cd82ce2bfaccf7ff247d4139d80f70156"} Jul 23 01:33:44.345858 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:44.345833 2572 generic.go:356] "Generic (PLEG): container finished" podID="c7b05400-d663-4df8-84e3-4689ddc165f0" containerID="5c6a2470a678e962005baf71cc4c0473ba2bf69a402bb37652e99a394bd9abb7" exitCode=0 Jul 23 01:33:44.345972 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:44.345887 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" event={"ID":"c7b05400-d663-4df8-84e3-4689ddc165f0","Type":"ContainerDied","Data":"5c6a2470a678e962005baf71cc4c0473ba2bf69a402bb37652e99a394bd9abb7"} Jul 23 01:33:46.243835 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.243811 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:46.315106 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.315083 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8v27\" (UniqueName: \"kubernetes.io/projected/c7b05400-d663-4df8-84e3-4689ddc165f0-kube-api-access-c8v27\") pod \"c7b05400-d663-4df8-84e3-4689ddc165f0\" (UID: \"c7b05400-d663-4df8-84e3-4689ddc165f0\") " Jul 23 01:33:46.315230 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.315168 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7b05400-d663-4df8-84e3-4689ddc165f0-bundle\") pod \"c7b05400-d663-4df8-84e3-4689ddc165f0\" (UID: \"c7b05400-d663-4df8-84e3-4689ddc165f0\") " Jul 23 01:33:46.315230 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.315203 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7b05400-d663-4df8-84e3-4689ddc165f0-util\") pod \"c7b05400-d663-4df8-84e3-4689ddc165f0\" (UID: \"c7b05400-d663-4df8-84e3-4689ddc165f0\") " Jul 23 01:33:46.315700 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.315666 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b05400-d663-4df8-84e3-4689ddc165f0-bundle" (OuterVolumeSpecName: "bundle") pod "c7b05400-d663-4df8-84e3-4689ddc165f0" (UID: "c7b05400-d663-4df8-84e3-4689ddc165f0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:33:46.317519 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.317494 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b05400-d663-4df8-84e3-4689ddc165f0-kube-api-access-c8v27" (OuterVolumeSpecName: "kube-api-access-c8v27") pod "c7b05400-d663-4df8-84e3-4689ddc165f0" (UID: "c7b05400-d663-4df8-84e3-4689ddc165f0"). InnerVolumeSpecName "kube-api-access-c8v27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:33:46.325521 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.325487 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b05400-d663-4df8-84e3-4689ddc165f0-util" (OuterVolumeSpecName: "util") pod "c7b05400-d663-4df8-84e3-4689ddc165f0" (UID: "c7b05400-d663-4df8-84e3-4689ddc165f0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:33:46.355867 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.355848 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" Jul 23 01:33:46.355984 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.355874 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb9tbc" event={"ID":"c7b05400-d663-4df8-84e3-4689ddc165f0","Type":"ContainerDied","Data":"0d3b13e7ebd980f2694fc9c81b5bc859c141e3003b9202a2b1326ed8570371a1"} Jul 23 01:33:46.355984 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.355928 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d3b13e7ebd980f2694fc9c81b5bc859c141e3003b9202a2b1326ed8570371a1" Jul 23 01:33:46.416049 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.416014 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7b05400-d663-4df8-84e3-4689ddc165f0-bundle\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:33:46.416049 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.416047 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7b05400-d663-4df8-84e3-4689ddc165f0-util\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:33:46.416239 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:46.416061 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8v27\" (UniqueName: \"kubernetes.io/projected/c7b05400-d663-4df8-84e3-4689ddc165f0-kube-api-access-c8v27\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:33:47.362558 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:47.362519 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fclxs" event={"ID":"7b4b55eb-77c5-41c2-9ccc-c9e785b5c7a5","Type":"ContainerStarted","Data":"d54ae198b3b368a436702b35f72f6ba0e81a24f14e3a99d19e00db044d61aa10"} Jul 23 01:33:47.377135 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:47.377083 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fclxs" podStartSLOduration=1.901852857 podStartE2EDuration="4.377069017s" totalImagesPullingTime="2.47521616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:33:47.376174915 +0000 UTC m=+339.921124089" watchObservedRunningTime="2026-07-23 01:33:47.377069017 +0000 UTC m=+339.922018190" Jul 23 01:33:52.243647 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.243606 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4cvhz"] Jul 23 01:33:52.244054 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.243911 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b05400-d663-4df8-84e3-4689ddc165f0" containerName="util" Jul 23 01:33:52.244054 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.243922 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b05400-d663-4df8-84e3-4689ddc165f0" containerName="util" Jul 23 01:33:52.244054 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.243931 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b05400-d663-4df8-84e3-4689ddc165f0" containerName="pull" Jul 23 01:33:52.244054 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.243936 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b05400-d663-4df8-84e3-4689ddc165f0" containerName="pull" Jul 23 01:33:52.244054 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.243947 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b05400-d663-4df8-84e3-4689ddc165f0" containerName="extract" Jul 23 01:33:52.244054 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.243954 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b05400-d663-4df8-84e3-4689ddc165f0" containerName="extract" Jul 23 01:33:52.244054 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.244001 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b05400-d663-4df8-84e3-4689ddc165f0" containerName="extract" Jul 23 01:33:52.246072 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.246055 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4cvhz" Jul 23 01:33:52.248562 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.248542 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-rccqv\"" Jul 23 01:33:52.256669 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.256622 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4cvhz"] Jul 23 01:33:52.360196 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.360159 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8694f1f4-1571-410c-a737-b02d856b6e76-bound-sa-token\") pod \"cert-manager-55b4d954d5-4cvhz\" (UID: \"8694f1f4-1571-410c-a737-b02d856b6e76\") " pod="cert-manager/cert-manager-55b4d954d5-4cvhz" Jul 23 01:33:52.360368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.360217 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2dc\" (UniqueName: \"kubernetes.io/projected/8694f1f4-1571-410c-a737-b02d856b6e76-kube-api-access-cc2dc\") pod \"cert-manager-55b4d954d5-4cvhz\" (UID: \"8694f1f4-1571-410c-a737-b02d856b6e76\") " pod="cert-manager/cert-manager-55b4d954d5-4cvhz" Jul 23 01:33:52.461544 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.461509 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8694f1f4-1571-410c-a737-b02d856b6e76-bound-sa-token\") pod \"cert-manager-55b4d954d5-4cvhz\" (UID: \"8694f1f4-1571-410c-a737-b02d856b6e76\") " pod="cert-manager/cert-manager-55b4d954d5-4cvhz" Jul 23 01:33:52.461726 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.461584 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2dc\" (UniqueName: \"kubernetes.io/projected/8694f1f4-1571-410c-a737-b02d856b6e76-kube-api-access-cc2dc\") pod \"cert-manager-55b4d954d5-4cvhz\" (UID: \"8694f1f4-1571-410c-a737-b02d856b6e76\") " pod="cert-manager/cert-manager-55b4d954d5-4cvhz" Jul 23 01:33:52.469422 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.469384 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8694f1f4-1571-410c-a737-b02d856b6e76-bound-sa-token\") pod \"cert-manager-55b4d954d5-4cvhz\" (UID: \"8694f1f4-1571-410c-a737-b02d856b6e76\") " pod="cert-manager/cert-manager-55b4d954d5-4cvhz" Jul 23 01:33:52.469566 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.469499 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2dc\" (UniqueName: \"kubernetes.io/projected/8694f1f4-1571-410c-a737-b02d856b6e76-kube-api-access-cc2dc\") pod \"cert-manager-55b4d954d5-4cvhz\" (UID: \"8694f1f4-1571-410c-a737-b02d856b6e76\") " pod="cert-manager/cert-manager-55b4d954d5-4cvhz" Jul 23 01:33:52.555390 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.555292 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4cvhz" Jul 23 01:33:52.692919 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:52.692847 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4cvhz"] Jul 23 01:33:52.695233 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:33:52.695204 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8694f1f4_1571_410c_a737_b02d856b6e76.slice/crio-66a3152605ada698b1b1dfa3973c89bd577724bdb9acc1bd402a08b98dfd9601 WatchSource:0}: Error finding container 66a3152605ada698b1b1dfa3973c89bd577724bdb9acc1bd402a08b98dfd9601: Status 404 returned error can't find the container with id 66a3152605ada698b1b1dfa3973c89bd577724bdb9acc1bd402a08b98dfd9601 Jul 23 01:33:53.387721 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:53.387686 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4cvhz" event={"ID":"8694f1f4-1571-410c-a737-b02d856b6e76","Type":"ContainerStarted","Data":"efdd78b9960cde8401407cef47a0f5edbc7714890224d5df8b0becd2502e5491"} Jul 23 01:33:53.387721 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:53.387727 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4cvhz" event={"ID":"8694f1f4-1571-410c-a737-b02d856b6e76","Type":"ContainerStarted","Data":"66a3152605ada698b1b1dfa3973c89bd577724bdb9acc1bd402a08b98dfd9601"} Jul 23 01:33:53.403347 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:53.403298 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4cvhz" podStartSLOduration=1.403283732 podStartE2EDuration="1.403283732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:33:53.402220268 +0000 UTC m=+345.947169440" watchObservedRunningTime="2026-07-23 01:33:53.403283732 +0000 UTC m=+345.948232906" Jul 23 01:33:57.218147 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.218108 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2"] Jul 23 01:33:57.220280 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.220263 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:33:57.222619 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.222589 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 01:33:57.222749 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.222607 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 01:33:57.222749 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.222715 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tqqgm\"" Jul 23 01:33:57.228467 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.228441 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2"] Jul 23 01:33:57.299790 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.299750 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2\" (UID: \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:33:57.299790 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.299791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2\" (UID: \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:33:57.300017 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.299816 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2cg\" (UniqueName: \"kubernetes.io/projected/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-kube-api-access-vx2cg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2\" (UID: \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:33:57.400206 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.400170 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2\" (UID: \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:33:57.400376 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.400220 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2\" (UID: \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:33:57.400376 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.400258 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2cg\" (UniqueName: \"kubernetes.io/projected/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-kube-api-access-vx2cg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2\" (UID: \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:33:57.400692 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.400670 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2\" (UID: \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:33:57.400764 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.400679 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2\" (UID: \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:33:57.408703 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.408679 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2cg\" (UniqueName: \"kubernetes.io/projected/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-kube-api-access-vx2cg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2\" (UID: \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:33:57.529953 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.529857 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:33:57.668692 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:57.668603 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2"] Jul 23 01:33:57.671055 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:33:57.671026 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6d4051_2f59_472d_a42f_13f4db1fbf7f.slice/crio-d0c07c463c426f1d5d62ab1432ba5834abeb5098f74aabc6a88e76731965f5f9 WatchSource:0}: Error finding container d0c07c463c426f1d5d62ab1432ba5834abeb5098f74aabc6a88e76731965f5f9: Status 404 returned error can't find the container with id d0c07c463c426f1d5d62ab1432ba5834abeb5098f74aabc6a88e76731965f5f9 Jul 23 01:33:58.409712 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:58.409677 2572 generic.go:356] "Generic (PLEG): container finished" podID="bc6d4051-2f59-472d-a42f-13f4db1fbf7f" containerID="51833d05c3ca3336e971271b7649b1884c660a80ef83384bb3d37411353b2608" exitCode=0 Jul 23 01:33:58.410076 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:58.409768 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" event={"ID":"bc6d4051-2f59-472d-a42f-13f4db1fbf7f","Type":"ContainerDied","Data":"51833d05c3ca3336e971271b7649b1884c660a80ef83384bb3d37411353b2608"} Jul 23 01:33:58.410076 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:33:58.409817 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" event={"ID":"bc6d4051-2f59-472d-a42f-13f4db1fbf7f","Type":"ContainerStarted","Data":"d0c07c463c426f1d5d62ab1432ba5834abeb5098f74aabc6a88e76731965f5f9"} Jul 23 01:34:00.420777 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:00.420688 2572 generic.go:356] "Generic (PLEG): container finished" podID="bc6d4051-2f59-472d-a42f-13f4db1fbf7f" containerID="d98968bb88cafdbff2735a46f8f3a4f3fadce573a8cd594844d8400229bc7f5c" exitCode=0 Jul 23 01:34:00.421334 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:00.420764 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" event={"ID":"bc6d4051-2f59-472d-a42f-13f4db1fbf7f","Type":"ContainerDied","Data":"d98968bb88cafdbff2735a46f8f3a4f3fadce573a8cd594844d8400229bc7f5c"} Jul 23 01:34:01.427114 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:01.427084 2572 generic.go:356] "Generic (PLEG): container finished" podID="bc6d4051-2f59-472d-a42f-13f4db1fbf7f" containerID="02d9e946d88e36e52361cbe3a10f1b8461db92ecd6cd8b3799df81b7424a468f" exitCode=0 Jul 23 01:34:01.427589 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:01.427154 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" event={"ID":"bc6d4051-2f59-472d-a42f-13f4db1fbf7f","Type":"ContainerDied","Data":"02d9e946d88e36e52361cbe3a10f1b8461db92ecd6cd8b3799df81b7424a468f"} Jul 23 01:34:02.562852 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:02.562825 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:34:02.643404 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:02.643366 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-bundle\") pod \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\" (UID: \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\") " Jul 23 01:34:02.643570 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:02.643462 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-util\") pod \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\" (UID: \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\") " Jul 23 01:34:02.643570 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:02.643494 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx2cg\" (UniqueName: \"kubernetes.io/projected/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-kube-api-access-vx2cg\") pod \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\" (UID: \"bc6d4051-2f59-472d-a42f-13f4db1fbf7f\") " Jul 23 01:34:02.644131 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:02.644100 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-bundle" (OuterVolumeSpecName: "bundle") pod "bc6d4051-2f59-472d-a42f-13f4db1fbf7f" (UID: "bc6d4051-2f59-472d-a42f-13f4db1fbf7f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:34:02.645574 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:02.645544 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-kube-api-access-vx2cg" (OuterVolumeSpecName: "kube-api-access-vx2cg") pod "bc6d4051-2f59-472d-a42f-13f4db1fbf7f" (UID: "bc6d4051-2f59-472d-a42f-13f4db1fbf7f"). InnerVolumeSpecName "kube-api-access-vx2cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:34:02.648972 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:02.648937 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-util" (OuterVolumeSpecName: "util") pod "bc6d4051-2f59-472d-a42f-13f4db1fbf7f" (UID: "bc6d4051-2f59-472d-a42f-13f4db1fbf7f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:34:02.744698 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:02.744585 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-bundle\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:34:02.744698 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:02.744618 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-util\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:34:02.744698 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:02.744628 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx2cg\" (UniqueName: \"kubernetes.io/projected/bc6d4051-2f59-472d-a42f-13f4db1fbf7f-kube-api-access-vx2cg\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:34:03.437820 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:03.437779 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" event={"ID":"bc6d4051-2f59-472d-a42f-13f4db1fbf7f","Type":"ContainerDied","Data":"d0c07c463c426f1d5d62ab1432ba5834abeb5098f74aabc6a88e76731965f5f9"} Jul 23 01:34:03.437994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:03.437831 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0c07c463c426f1d5d62ab1432ba5834abeb5098f74aabc6a88e76731965f5f9" Jul 23 01:34:03.437994 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:03.437793 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjdgf2" Jul 23 01:34:10.724177 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.724143 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr"] Jul 23 01:34:10.724657 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.724404 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6d4051-2f59-472d-a42f-13f4db1fbf7f" containerName="extract" Jul 23 01:34:10.724657 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.724415 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6d4051-2f59-472d-a42f-13f4db1fbf7f" containerName="extract" Jul 23 01:34:10.724657 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.724436 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6d4051-2f59-472d-a42f-13f4db1fbf7f" containerName="util" Jul 23 01:34:10.724657 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.724442 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6d4051-2f59-472d-a42f-13f4db1fbf7f" containerName="util" Jul 23 01:34:10.724657 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.724452 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6d4051-2f59-472d-a42f-13f4db1fbf7f" containerName="pull" Jul 23 01:34:10.724657 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.724458 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6d4051-2f59-472d-a42f-13f4db1fbf7f" containerName="pull" Jul 23 01:34:10.724657 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.724500 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc6d4051-2f59-472d-a42f-13f4db1fbf7f" containerName="extract" Jul 23 01:34:10.729866 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.729842 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.732517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.732480 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 01:34:10.732517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.732480 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 01:34:10.732517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.732513 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 01:34:10.732766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.732582 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zdsb9\"" Jul 23 01:34:10.732766 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.732585 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 01:34:10.740272 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.740250 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr"] Jul 23 01:34:10.810487 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.810440 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ea2a247-3878-47bf-b76c-e0749a73a665-tmp\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.810719 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.810557 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7mr5\" (UniqueName: \"kubernetes.io/projected/7ea2a247-3878-47bf-b76c-e0749a73a665-kube-api-access-b7mr5\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.810719 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.810586 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ea2a247-3878-47bf-b76c-e0749a73a665-manifests\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.810719 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.810625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ea2a247-3878-47bf-b76c-e0749a73a665-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.810719 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.810697 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ea2a247-3878-47bf-b76c-e0749a73a665-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.911787 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.911746 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7mr5\" (UniqueName: \"kubernetes.io/projected/7ea2a247-3878-47bf-b76c-e0749a73a665-kube-api-access-b7mr5\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.911787 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.911786 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ea2a247-3878-47bf-b76c-e0749a73a665-manifests\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.912056 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.911808 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ea2a247-3878-47bf-b76c-e0749a73a665-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.912056 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.911828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ea2a247-3878-47bf-b76c-e0749a73a665-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.912056 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.911896 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ea2a247-3878-47bf-b76c-e0749a73a665-tmp\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.912233 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.912207 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ea2a247-3878-47bf-b76c-e0749a73a665-manifests\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.914238 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.914209 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ea2a247-3878-47bf-b76c-e0749a73a665-tmp\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.914393 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.914377 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ea2a247-3878-47bf-b76c-e0749a73a665-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.914449 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.914433 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ea2a247-3878-47bf-b76c-e0749a73a665-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:10.919664 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:10.919626 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7mr5\" (UniqueName: \"kubernetes.io/projected/7ea2a247-3878-47bf-b76c-e0749a73a665-kube-api-access-b7mr5\") pod \"opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr\" (UID: \"7ea2a247-3878-47bf-b76c-e0749a73a665\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:11.040248 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:11.040159 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:11.192850 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:11.192819 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr"] Jul 23 01:34:11.196097 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:34:11.196065 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea2a247_3878_47bf_b76c_e0749a73a665.slice/crio-7c286e4226e54ef94684d88b684a16aaa6a1f1733abc1444a863a1b380795741 WatchSource:0}: Error finding container 7c286e4226e54ef94684d88b684a16aaa6a1f1733abc1444a863a1b380795741: Status 404 returned error can't find the container with id 7c286e4226e54ef94684d88b684a16aaa6a1f1733abc1444a863a1b380795741 Jul 23 01:34:11.476112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:11.476073 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" event={"ID":"7ea2a247-3878-47bf-b76c-e0749a73a665","Type":"ContainerStarted","Data":"7c286e4226e54ef94684d88b684a16aaa6a1f1733abc1444a863a1b380795741"} Jul 23 01:34:14.494624 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:14.494586 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" event={"ID":"7ea2a247-3878-47bf-b76c-e0749a73a665","Type":"ContainerStarted","Data":"48e04b3e7a27112e6a19b2cc120e2776d72b176212fa9691d544b60071f97d36"} Jul 23 01:34:15.500290 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:15.500259 2572 generic.go:356] "Generic (PLEG): container finished" podID="7ea2a247-3878-47bf-b76c-e0749a73a665" containerID="48e04b3e7a27112e6a19b2cc120e2776d72b176212fa9691d544b60071f97d36" exitCode=0 Jul 23 01:34:15.500667 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:15.500330 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" event={"ID":"7ea2a247-3878-47bf-b76c-e0749a73a665","Type":"ContainerDied","Data":"48e04b3e7a27112e6a19b2cc120e2776d72b176212fa9691d544b60071f97d36"} Jul 23 01:34:16.507107 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:16.507069 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" event={"ID":"7ea2a247-3878-47bf-b76c-e0749a73a665","Type":"ContainerStarted","Data":"0dccc19ffa6b98973bcde785a9765a4c68e1faa3dff4b30f16f22737da108225"} Jul 23 01:34:16.507500 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:16.507185 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:16.526745 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:16.526699 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" podStartSLOduration=1.9224668070000002 podStartE2EDuration="6.526685359s" totalImagesPullingTime="3.510526564s" totalInitContainerRuntime="1.093691988s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:34:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:34:16.525559276 +0000 UTC m=+369.070508449" watchObservedRunningTime="2026-07-23 01:34:16.526685359 +0000 UTC m=+369.071634531" Jul 23 01:34:27.513505 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:27.513472 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr" Jul 23 01:34:30.509353 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.509313 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85497ff65b-wnv44"] Jul 23 01:34:30.512424 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.512405 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" Jul 23 01:34:30.514870 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.514847 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 01:34:30.515010 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.514881 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 01:34:30.515010 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.514965 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 01:34:30.515010 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.514979 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cjnxx\"" Jul 23 01:34:30.515731 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.515715 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 01:34:30.521907 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.521880 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85497ff65b-wnv44"] Jul 23 01:34:30.648368 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.648323 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xkv9t"] Jul 23 01:34:30.651995 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.651972 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" Jul 23 01:34:30.655964 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.655943 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 23 01:34:30.656120 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.655948 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pj64x\"" Jul 23 01:34:30.663030 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.663004 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xkv9t"] Jul 23 01:34:30.677384 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.677333 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa85ab52-95a8-4d0f-88e3-a9d4d261524f-tmp\") pod \"kube-auth-proxy-85497ff65b-wnv44\" (UID: \"aa85ab52-95a8-4d0f-88e3-a9d4d261524f\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" Jul 23 01:34:30.677529 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.677426 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qszz\" (UniqueName: \"kubernetes.io/projected/aa85ab52-95a8-4d0f-88e3-a9d4d261524f-kube-api-access-4qszz\") pod \"kube-auth-proxy-85497ff65b-wnv44\" (UID: \"aa85ab52-95a8-4d0f-88e3-a9d4d261524f\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" Jul 23 01:34:30.677529 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.677477 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa85ab52-95a8-4d0f-88e3-a9d4d261524f-tls-certs\") pod \"kube-auth-proxy-85497ff65b-wnv44\" (UID: \"aa85ab52-95a8-4d0f-88e3-a9d4d261524f\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" Jul 23 01:34:30.778074 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.777985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa85ab52-95a8-4d0f-88e3-a9d4d261524f-tmp\") pod \"kube-auth-proxy-85497ff65b-wnv44\" (UID: \"aa85ab52-95a8-4d0f-88e3-a9d4d261524f\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" Jul 23 01:34:30.778074 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.778032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22f21c1f-dbd9-4a70-9606-0217f81be9fb-cert\") pod \"odh-model-controller-7998c67dc4-xkv9t\" (UID: \"22f21c1f-dbd9-4a70-9606-0217f81be9fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" Jul 23 01:34:30.778074 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.778053 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qszz\" (UniqueName: \"kubernetes.io/projected/aa85ab52-95a8-4d0f-88e3-a9d4d261524f-kube-api-access-4qszz\") pod \"kube-auth-proxy-85497ff65b-wnv44\" (UID: \"aa85ab52-95a8-4d0f-88e3-a9d4d261524f\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" Jul 23 01:34:30.778333 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.778084 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa85ab52-95a8-4d0f-88e3-a9d4d261524f-tls-certs\") pod \"kube-auth-proxy-85497ff65b-wnv44\" (UID: \"aa85ab52-95a8-4d0f-88e3-a9d4d261524f\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" Jul 23 01:34:30.778333 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.778117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f629j\" (UniqueName: \"kubernetes.io/projected/22f21c1f-dbd9-4a70-9606-0217f81be9fb-kube-api-access-f629j\") pod \"odh-model-controller-7998c67dc4-xkv9t\" (UID: \"22f21c1f-dbd9-4a70-9606-0217f81be9fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" Jul 23 01:34:30.781107 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.781074 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa85ab52-95a8-4d0f-88e3-a9d4d261524f-tmp\") pod \"kube-auth-proxy-85497ff65b-wnv44\" (UID: \"aa85ab52-95a8-4d0f-88e3-a9d4d261524f\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" Jul 23 01:34:30.781296 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.781271 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa85ab52-95a8-4d0f-88e3-a9d4d261524f-tls-certs\") pod \"kube-auth-proxy-85497ff65b-wnv44\" (UID: \"aa85ab52-95a8-4d0f-88e3-a9d4d261524f\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" Jul 23 01:34:30.790837 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.790811 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qszz\" (UniqueName: \"kubernetes.io/projected/aa85ab52-95a8-4d0f-88e3-a9d4d261524f-kube-api-access-4qszz\") pod \"kube-auth-proxy-85497ff65b-wnv44\" (UID: \"aa85ab52-95a8-4d0f-88e3-a9d4d261524f\") " pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" Jul 23 01:34:30.822974 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.822946 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" Jul 23 01:34:30.879541 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.879497 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22f21c1f-dbd9-4a70-9606-0217f81be9fb-cert\") pod \"odh-model-controller-7998c67dc4-xkv9t\" (UID: \"22f21c1f-dbd9-4a70-9606-0217f81be9fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" Jul 23 01:34:30.879749 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.879569 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f629j\" (UniqueName: \"kubernetes.io/projected/22f21c1f-dbd9-4a70-9606-0217f81be9fb-kube-api-access-f629j\") pod \"odh-model-controller-7998c67dc4-xkv9t\" (UID: \"22f21c1f-dbd9-4a70-9606-0217f81be9fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" Jul 23 01:34:30.879749 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:34:30.879690 2572 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 23 01:34:30.879867 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:34:30.879762 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f21c1f-dbd9-4a70-9606-0217f81be9fb-cert podName:22f21c1f-dbd9-4a70-9606-0217f81be9fb nodeName:}" failed. No retries permitted until 2026-07-23 01:34:31.379741914 +0000 UTC m=+383.924691079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/22f21c1f-dbd9-4a70-9606-0217f81be9fb-cert") pod "odh-model-controller-7998c67dc4-xkv9t" (UID: "22f21c1f-dbd9-4a70-9606-0217f81be9fb") : secret "odh-model-controller-webhook-cert" not found Jul 23 01:34:30.890081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.890051 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f629j\" (UniqueName: \"kubernetes.io/projected/22f21c1f-dbd9-4a70-9606-0217f81be9fb-kube-api-access-f629j\") pod \"odh-model-controller-7998c67dc4-xkv9t\" (UID: \"22f21c1f-dbd9-4a70-9606-0217f81be9fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" Jul 23 01:34:30.976836 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:30.976546 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85497ff65b-wnv44"] Jul 23 01:34:30.979467 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:34:30.979438 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa85ab52_95a8_4d0f_88e3_a9d4d261524f.slice/crio-dcecd71c3dd915468105aa7969e27c2fa5a6653a7bc7e1c0b462dd26e7a5415e WatchSource:0}: Error finding container dcecd71c3dd915468105aa7969e27c2fa5a6653a7bc7e1c0b462dd26e7a5415e: Status 404 returned error can't find the container with id dcecd71c3dd915468105aa7969e27c2fa5a6653a7bc7e1c0b462dd26e7a5415e Jul 23 01:34:31.384353 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.384319 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22f21c1f-dbd9-4a70-9606-0217f81be9fb-cert\") pod \"odh-model-controller-7998c67dc4-xkv9t\" (UID: \"22f21c1f-dbd9-4a70-9606-0217f81be9fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" Jul 23 01:34:31.386709 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.386683 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22f21c1f-dbd9-4a70-9606-0217f81be9fb-cert\") pod \"odh-model-controller-7998c67dc4-xkv9t\" (UID: \"22f21c1f-dbd9-4a70-9606-0217f81be9fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" Jul 23 01:34:31.496343 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.496302 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tf2sm"] Jul 23 01:34:31.500750 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.500726 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" Jul 23 01:34:31.503168 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.503138 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 23 01:34:31.503295 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.503272 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-8sqzj\"" Jul 23 01:34:31.506696 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.506672 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tf2sm"] Jul 23 01:34:31.564928 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.564893 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" Jul 23 01:34:31.577002 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.576966 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" event={"ID":"aa85ab52-95a8-4d0f-88e3-a9d4d261524f","Type":"ContainerStarted","Data":"dcecd71c3dd915468105aa7969e27c2fa5a6653a7bc7e1c0b462dd26e7a5415e"} Jul 23 01:34:31.659240 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.658575 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6"] Jul 23 01:34:31.662570 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.662547 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.666213 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.665991 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 01:34:31.666213 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.665991 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 01:34:31.666213 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.666130 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 01:34:31.668726 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.668703 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d6fkq\"" Jul 23 01:34:31.668965 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.668947 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 01:34:31.683291 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.682713 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6"] Jul 23 01:34:31.686536 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.686510 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw6nn\" (UniqueName: \"kubernetes.io/projected/d5265802-05d7-42fa-865f-70d0f15023ee-kube-api-access-hw6nn\") pod \"model-serving-api-54d89bd5fc-tf2sm\" (UID: \"d5265802-05d7-42fa-865f-70d0f15023ee\") " pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" Jul 23 01:34:31.686859 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.686830 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5265802-05d7-42fa-865f-70d0f15023ee-tls-certs\") pod \"model-serving-api-54d89bd5fc-tf2sm\" (UID: \"d5265802-05d7-42fa-865f-70d0f15023ee\") " pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" Jul 23 01:34:31.723807 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.723678 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xkv9t"] Jul 23 01:34:31.726318 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:34:31.726289 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f21c1f_dbd9_4a70_9606_0217f81be9fb.slice/crio-237c3b4ee340470d5793e7da9f0215586a762474338b52cd1785ef8d24923419 WatchSource:0}: Error finding container 237c3b4ee340470d5793e7da9f0215586a762474338b52cd1785ef8d24923419: Status 404 returned error can't find the container with id 237c3b4ee340470d5793e7da9f0215586a762474338b52cd1785ef8d24923419 Jul 23 01:34:31.787997 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.787957 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c07c9a-4470-4eca-93a1-9d03652382be-istio-token\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.788185 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.788011 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rz2p\" (UniqueName: \"kubernetes.io/projected/67c07c9a-4470-4eca-93a1-9d03652382be-kube-api-access-4rz2p\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.788185 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.788049 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw6nn\" (UniqueName: \"kubernetes.io/projected/d5265802-05d7-42fa-865f-70d0f15023ee-kube-api-access-hw6nn\") pod \"model-serving-api-54d89bd5fc-tf2sm\" (UID: \"d5265802-05d7-42fa-865f-70d0f15023ee\") " pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" Jul 23 01:34:31.788185 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.788083 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.788185 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.788141 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.788185 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.788169 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-cacerts\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.788421 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.788204 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5265802-05d7-42fa-865f-70d0f15023ee-tls-certs\") pod \"model-serving-api-54d89bd5fc-tf2sm\" (UID: \"d5265802-05d7-42fa-865f-70d0f15023ee\") " pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" Jul 23 01:34:31.788421 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.788235 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c07c9a-4470-4eca-93a1-9d03652382be-local-certs\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.788421 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.788273 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67c07c9a-4470-4eca-93a1-9d03652382be-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.791304 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.791277 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5265802-05d7-42fa-865f-70d0f15023ee-tls-certs\") pod \"model-serving-api-54d89bd5fc-tf2sm\" (UID: \"d5265802-05d7-42fa-865f-70d0f15023ee\") " pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" Jul 23 01:34:31.799189 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.799164 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw6nn\" (UniqueName: \"kubernetes.io/projected/d5265802-05d7-42fa-865f-70d0f15023ee-kube-api-access-hw6nn\") pod \"model-serving-api-54d89bd5fc-tf2sm\" (UID: \"d5265802-05d7-42fa-865f-70d0f15023ee\") " pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" Jul 23 01:34:31.812871 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.812839 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" Jul 23 01:34:31.889521 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.889486 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c07c9a-4470-4eca-93a1-9d03652382be-local-certs\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.889683 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.889532 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67c07c9a-4470-4eca-93a1-9d03652382be-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.889683 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.889561 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c07c9a-4470-4eca-93a1-9d03652382be-istio-token\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.889683 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.889581 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rz2p\" (UniqueName: \"kubernetes.io/projected/67c07c9a-4470-4eca-93a1-9d03652382be-kube-api-access-4rz2p\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.889845 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.889750 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.889897 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.889845 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.889897 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.889878 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-cacerts\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.890283 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.890254 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67c07c9a-4470-4eca-93a1-9d03652382be-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.891903 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.891873 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c07c9a-4470-4eca-93a1-9d03652382be-local-certs\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.892181 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.892155 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.892265 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.892164 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-cacerts\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.892411 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.892396 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.903998 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.903970 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rz2p\" (UniqueName: \"kubernetes.io/projected/67c07c9a-4470-4eca-93a1-9d03652382be-kube-api-access-4rz2p\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.906158 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.906100 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c07c9a-4470-4eca-93a1-9d03652382be-istio-token\") pod \"istiod-openshift-gateway-7574459656-l4lr6\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:31.953792 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.953755 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tf2sm"] Jul 23 01:34:31.957236 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:34:31.957201 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5265802_05d7_42fa_865f_70d0f15023ee.slice/crio-09ffc74b35be0857ec8883d181a882442be31300a881a883e568b949e89add6c WatchSource:0}: Error finding container 09ffc74b35be0857ec8883d181a882442be31300a881a883e568b949e89add6c: Status 404 returned error can't find the container with id 09ffc74b35be0857ec8883d181a882442be31300a881a883e568b949e89add6c Jul 23 01:34:31.990649 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:31.990597 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:32.162880 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:32.162787 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6"] Jul 23 01:34:32.164801 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:34:32.164769 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c07c9a_4470_4eca_93a1_9d03652382be.slice/crio-1a735be9e2a6d0e8aa98e71bc57325b0a93f882dd6af3e7ba7f94d93e24d850a WatchSource:0}: Error finding container 1a735be9e2a6d0e8aa98e71bc57325b0a93f882dd6af3e7ba7f94d93e24d850a: Status 404 returned error can't find the container with id 1a735be9e2a6d0e8aa98e71bc57325b0a93f882dd6af3e7ba7f94d93e24d850a Jul 23 01:34:32.591738 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:32.591628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" event={"ID":"22f21c1f-dbd9-4a70-9606-0217f81be9fb","Type":"ContainerStarted","Data":"237c3b4ee340470d5793e7da9f0215586a762474338b52cd1785ef8d24923419"} Jul 23 01:34:32.596484 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:32.596391 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" event={"ID":"d5265802-05d7-42fa-865f-70d0f15023ee","Type":"ContainerStarted","Data":"09ffc74b35be0857ec8883d181a882442be31300a881a883e568b949e89add6c"} Jul 23 01:34:32.598577 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:32.598499 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" event={"ID":"67c07c9a-4470-4eca-93a1-9d03652382be","Type":"ContainerStarted","Data":"1a735be9e2a6d0e8aa98e71bc57325b0a93f882dd6af3e7ba7f94d93e24d850a"} Jul 23 01:34:36.951586 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:36.951548 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 01:34:36.952019 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:36.951613 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 01:34:37.632457 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:37.632412 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" event={"ID":"67c07c9a-4470-4eca-93a1-9d03652382be","Type":"ContainerStarted","Data":"205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c"} Jul 23 01:34:37.632737 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:37.632708 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:37.634394 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:37.634361 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" event={"ID":"aa85ab52-95a8-4d0f-88e3-a9d4d261524f","Type":"ContainerStarted","Data":"2a83e804483320a5392dc17e66b55ab647e3565f37d0bb155a003749ab96cb26"} Jul 23 01:34:37.636371 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:37.636263 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" event={"ID":"22f21c1f-dbd9-4a70-9606-0217f81be9fb","Type":"ContainerStarted","Data":"59414f533b85c82c626c75dee4f1fd01623f3da5d5c72a3da0842d3a1e2ff54e"} Jul 23 01:34:37.636371 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:37.636367 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" Jul 23 01:34:37.638942 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:37.638917 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" event={"ID":"d5265802-05d7-42fa-865f-70d0f15023ee","Type":"ContainerStarted","Data":"ec4a40bf7bb9cfaacd26d3c35ad87ee6e4218aedf21a50e0c19686c8351dbd49"} Jul 23 01:34:37.639112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:37.639089 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" Jul 23 01:34:37.654647 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:37.654575 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" podStartSLOduration=1.871080467 podStartE2EDuration="6.654556603s" totalImagesPullingTime="4.783476136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:34:37.651448996 +0000 UTC m=+390.196398200" watchObservedRunningTime="2026-07-23 01:34:37.654556603 +0000 UTC m=+390.199505775" Jul 23 01:34:37.667251 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:37.667202 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" podStartSLOduration=2.434248166 podStartE2EDuration="7.667188501s" totalImagesPullingTime="5.232940335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:34:37.665523032 +0000 UTC m=+390.210472206" watchObservedRunningTime="2026-07-23 01:34:37.667188501 +0000 UTC m=+390.212137674" Jul 23 01:34:37.679850 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:37.679804 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85497ff65b-wnv44" podStartSLOduration=1.716328233 podStartE2EDuration="7.679790265s" totalImagesPullingTime="5.963462032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:34:37.679477303 +0000 UTC m=+390.224426476" watchObservedRunningTime="2026-07-23 01:34:37.679790265 +0000 UTC m=+390.224739471" Jul 23 01:34:37.694149 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:37.694086 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" podStartSLOduration=1.708901612 podStartE2EDuration="6.694066286s" totalImagesPullingTime="4.985164674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:34:37.693569468 +0000 UTC m=+390.238518664" watchObservedRunningTime="2026-07-23 01:34:37.694066286 +0000 UTC m=+390.239015458" Jul 23 01:34:38.645834 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:38.645802 2572 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-l4lr6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 01:34:38.646211 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:38.645852 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" podUID="67c07c9a-4470-4eca-93a1-9d03652382be" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 01:34:39.271442 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.271402 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd"] Jul 23 01:34:39.274775 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.274757 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.296300 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.296274 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd"] Jul 23 01:34:39.363887 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.363847 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b7b7\" (UniqueName: \"kubernetes.io/projected/6adbbd58-4e62-4244-9840-970e0b9d740a-kube-api-access-9b7b7\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.364059 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.363901 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6adbbd58-4e62-4244-9840-970e0b9d740a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.364059 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.363970 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6adbbd58-4e62-4244-9840-970e0b9d740a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.364059 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.363997 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6adbbd58-4e62-4244-9840-970e0b9d740a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.364059 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.364024 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adbbd58-4e62-4244-9840-970e0b9d740a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.364059 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.364050 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adbbd58-4e62-4244-9840-970e0b9d740a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.364220 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.364083 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6adbbd58-4e62-4244-9840-970e0b9d740a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.464513 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.464477 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6adbbd58-4e62-4244-9840-970e0b9d740a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.464513 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.464517 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6adbbd58-4e62-4244-9840-970e0b9d740a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.464798 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.464534 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6adbbd58-4e62-4244-9840-970e0b9d740a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.464798 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.464556 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adbbd58-4e62-4244-9840-970e0b9d740a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.464798 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.464581 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adbbd58-4e62-4244-9840-970e0b9d740a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.464798 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.464624 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6adbbd58-4e62-4244-9840-970e0b9d740a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.464798 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.464685 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b7b7\" (UniqueName: \"kubernetes.io/projected/6adbbd58-4e62-4244-9840-970e0b9d740a-kube-api-access-9b7b7\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.465451 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.465427 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6adbbd58-4e62-4244-9840-970e0b9d740a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.466989 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.466962 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adbbd58-4e62-4244-9840-970e0b9d740a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.467118 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.467064 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6adbbd58-4e62-4244-9840-970e0b9d740a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.467359 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.467339 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6adbbd58-4e62-4244-9840-970e0b9d740a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.467449 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.467428 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6adbbd58-4e62-4244-9840-970e0b9d740a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.473827 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.473802 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b7b7\" (UniqueName: \"kubernetes.io/projected/6adbbd58-4e62-4244-9840-970e0b9d740a-kube-api-access-9b7b7\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.474067 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.474044 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adbbd58-4e62-4244-9840-970e0b9d740a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-gnjxd\" (UID: \"6adbbd58-4e62-4244-9840-970e0b9d740a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.583811 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.583721 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:39.735375 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.735347 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd"] Jul 23 01:34:39.737969 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:34:39.737933 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adbbd58_4e62_4244_9840_970e0b9d740a.slice/crio-bbbd911083185bbe64652933efdbd70452b60455261fb90ac53cb2efdc24144d WatchSource:0}: Error finding container bbbd911083185bbe64652933efdbd70452b60455261fb90ac53cb2efdc24144d: Status 404 returned error can't find the container with id bbbd911083185bbe64652933efdbd70452b60455261fb90ac53cb2efdc24144d Jul 23 01:34:39.742123 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.742085 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 01:34:39.742240 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:39.742160 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 01:34:40.656521 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:40.656480 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" event={"ID":"6adbbd58-4e62-4244-9840-970e0b9d740a","Type":"ContainerStarted","Data":"65b08dacff46afd3f75a3a6171476b6427402a677db95331243065cf9958c648"} Jul 23 01:34:40.656521 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:40.656529 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" event={"ID":"6adbbd58-4e62-4244-9840-970e0b9d740a","Type":"ContainerStarted","Data":"bbbd911083185bbe64652933efdbd70452b60455261fb90ac53cb2efdc24144d"} Jul 23 01:34:40.656806 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:40.656679 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:40.658421 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:40.658399 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" Jul 23 01:34:40.700223 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:40.700175 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gnjxd" podStartSLOduration=1.700159517 podStartE2EDuration="1.700159517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:34:40.696894426 +0000 UTC m=+393.241843652" watchObservedRunningTime="2026-07-23 01:34:40.700159517 +0000 UTC m=+393.245108692" Jul 23 01:34:40.777751 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:40.777721 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6"] Jul 23 01:34:40.778141 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:40.777878 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" podUID="67c07c9a-4470-4eca-93a1-9d03652382be" containerName="discovery" containerID="cri-o://205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c" gracePeriod=30 Jul 23 01:34:40.778682 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:40.778659 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:41.035528 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.035497 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:41.079127 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.079050 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rz2p\" (UniqueName: \"kubernetes.io/projected/67c07c9a-4470-4eca-93a1-9d03652382be-kube-api-access-4rz2p\") pod \"67c07c9a-4470-4eca-93a1-9d03652382be\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " Jul 23 01:34:41.079287 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.079143 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-cacerts\") pod \"67c07c9a-4470-4eca-93a1-9d03652382be\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " Jul 23 01:34:41.079287 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.079171 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c07c9a-4470-4eca-93a1-9d03652382be-local-certs\") pod \"67c07c9a-4470-4eca-93a1-9d03652382be\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " Jul 23 01:34:41.079287 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.079231 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c07c9a-4470-4eca-93a1-9d03652382be-istio-token\") pod \"67c07c9a-4470-4eca-93a1-9d03652382be\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " Jul 23 01:34:41.079287 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.079256 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-istio-kubeconfig\") pod \"67c07c9a-4470-4eca-93a1-9d03652382be\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " Jul 23 01:34:41.079571 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.079292 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-istio-csr-dns-cert\") pod \"67c07c9a-4470-4eca-93a1-9d03652382be\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " Jul 23 01:34:41.079571 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.079321 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67c07c9a-4470-4eca-93a1-9d03652382be-istio-csr-ca-configmap\") pod \"67c07c9a-4470-4eca-93a1-9d03652382be\" (UID: \"67c07c9a-4470-4eca-93a1-9d03652382be\") " Jul 23 01:34:41.079945 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.079916 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c07c9a-4470-4eca-93a1-9d03652382be-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "67c07c9a-4470-4eca-93a1-9d03652382be" (UID: "67c07c9a-4470-4eca-93a1-9d03652382be"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:34:41.081903 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.081867 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "67c07c9a-4470-4eca-93a1-9d03652382be" (UID: "67c07c9a-4470-4eca-93a1-9d03652382be"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:34:41.082148 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.082121 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c07c9a-4470-4eca-93a1-9d03652382be-kube-api-access-4rz2p" (OuterVolumeSpecName: "kube-api-access-4rz2p") pod "67c07c9a-4470-4eca-93a1-9d03652382be" (UID: "67c07c9a-4470-4eca-93a1-9d03652382be"). InnerVolumeSpecName "kube-api-access-4rz2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:34:41.082496 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.082471 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-cacerts" (OuterVolumeSpecName: "cacerts") pod "67c07c9a-4470-4eca-93a1-9d03652382be" (UID: "67c07c9a-4470-4eca-93a1-9d03652382be"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:34:41.082496 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.082474 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c07c9a-4470-4eca-93a1-9d03652382be-istio-token" (OuterVolumeSpecName: "istio-token") pod "67c07c9a-4470-4eca-93a1-9d03652382be" (UID: "67c07c9a-4470-4eca-93a1-9d03652382be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:34:41.082611 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.082491 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "67c07c9a-4470-4eca-93a1-9d03652382be" (UID: "67c07c9a-4470-4eca-93a1-9d03652382be"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:34:41.083292 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.082934 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c07c9a-4470-4eca-93a1-9d03652382be-local-certs" (OuterVolumeSpecName: "local-certs") pod "67c07c9a-4470-4eca-93a1-9d03652382be" (UID: "67c07c9a-4470-4eca-93a1-9d03652382be"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:34:41.180765 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.180730 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rz2p\" (UniqueName: \"kubernetes.io/projected/67c07c9a-4470-4eca-93a1-9d03652382be-kube-api-access-4rz2p\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:34:41.180765 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.180759 2572 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-cacerts\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:34:41.180765 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.180769 2572 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c07c9a-4470-4eca-93a1-9d03652382be-local-certs\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:34:41.180995 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.180778 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c07c9a-4470-4eca-93a1-9d03652382be-istio-token\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:34:41.180995 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.180786 2572 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-istio-kubeconfig\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:34:41.180995 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.180793 2572 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67c07c9a-4470-4eca-93a1-9d03652382be-istio-csr-dns-cert\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:34:41.180995 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.180802 2572 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67c07c9a-4470-4eca-93a1-9d03652382be-istio-csr-ca-configmap\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:34:41.663620 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.663588 2572 generic.go:356] "Generic (PLEG): container finished" podID="67c07c9a-4470-4eca-93a1-9d03652382be" containerID="205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c" exitCode=0 Jul 23 01:34:41.663898 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.663626 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" event={"ID":"67c07c9a-4470-4eca-93a1-9d03652382be","Type":"ContainerDied","Data":"205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c"} Jul 23 01:34:41.663898 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.663672 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" Jul 23 01:34:41.663898 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.663705 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6" event={"ID":"67c07c9a-4470-4eca-93a1-9d03652382be","Type":"ContainerDied","Data":"1a735be9e2a6d0e8aa98e71bc57325b0a93f882dd6af3e7ba7f94d93e24d850a"} Jul 23 01:34:41.663898 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.663732 2572 scope.go:117] "RemoveContainer" containerID="205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c" Jul 23 01:34:41.673008 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.672980 2572 scope.go:117] "RemoveContainer" containerID="205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c" Jul 23 01:34:41.673283 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:34:41.673262 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c\": container with ID starting with 205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c not found: ID does not exist" containerID="205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c" Jul 23 01:34:41.673351 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.673290 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c"} err="failed to get container status \"205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c\": rpc error: code = NotFound desc = could not find container \"205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c\": container with ID starting with 205f20951a9aa44f18fc02158abb1ebf7a0caf565b28125dbfafd0578f1eb75c not found: ID does not exist" Jul 23 01:34:41.688610 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.688583 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6"] Jul 23 01:34:41.695528 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.695503 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l4lr6"] Jul 23 01:34:41.764069 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.764040 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl"] Jul 23 01:34:41.764378 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.764358 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67c07c9a-4470-4eca-93a1-9d03652382be" containerName="discovery" Jul 23 01:34:41.764378 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.764376 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="67c07c9a-4470-4eca-93a1-9d03652382be" containerName="discovery" Jul 23 01:34:41.764533 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.764469 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="67c07c9a-4470-4eca-93a1-9d03652382be" containerName="discovery" Jul 23 01:34:41.768715 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.768695 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" Jul 23 01:34:41.771693 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.771187 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-kjll6\"" Jul 23 01:34:41.771693 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.771329 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 23 01:34:41.785504 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.778818 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws"] Jul 23 01:34:41.790723 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.790661 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" Jul 23 01:34:41.793081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.792098 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl"] Jul 23 01:34:41.793081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.793066 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-74qj7\"" Jul 23 01:34:41.794083 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.793439 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 23 01:34:41.794563 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.794517 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws"] Jul 23 01:34:41.890035 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.889997 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/787d4d5b-c973-4a3c-956b-c53163ef2b3a-cert\") pod \"kserve-controller-manager-d6f4b65b8-cfkpl\" (UID: \"787d4d5b-c973-4a3c-956b-c53163ef2b3a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" Jul 23 01:34:41.890035 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.890043 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c96e9f7-295f-401d-8459-b336023730c3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-258ws\" (UID: \"8c96e9f7-295f-401d-8459-b336023730c3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" Jul 23 01:34:41.890281 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.890060 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc6wk\" (UniqueName: \"kubernetes.io/projected/8c96e9f7-295f-401d-8459-b336023730c3-kube-api-access-wc6wk\") pod \"llmisvc-controller-manager-777c6bd7d5-258ws\" (UID: \"8c96e9f7-295f-401d-8459-b336023730c3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" Jul 23 01:34:41.890281 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.890122 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzn5t\" (UniqueName: \"kubernetes.io/projected/787d4d5b-c973-4a3c-956b-c53163ef2b3a-kube-api-access-gzn5t\") pod \"kserve-controller-manager-d6f4b65b8-cfkpl\" (UID: \"787d4d5b-c973-4a3c-956b-c53163ef2b3a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" Jul 23 01:34:41.981504 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.981412 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67c07c9a-4470-4eca-93a1-9d03652382be" path="/var/lib/kubelet/pods/67c07c9a-4470-4eca-93a1-9d03652382be/volumes" Jul 23 01:34:41.991093 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.991061 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c96e9f7-295f-401d-8459-b336023730c3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-258ws\" (UID: \"8c96e9f7-295f-401d-8459-b336023730c3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" Jul 23 01:34:41.991093 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.991104 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc6wk\" (UniqueName: \"kubernetes.io/projected/8c96e9f7-295f-401d-8459-b336023730c3-kube-api-access-wc6wk\") pod \"llmisvc-controller-manager-777c6bd7d5-258ws\" (UID: \"8c96e9f7-295f-401d-8459-b336023730c3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" Jul 23 01:34:41.991319 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.991152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzn5t\" (UniqueName: \"kubernetes.io/projected/787d4d5b-c973-4a3c-956b-c53163ef2b3a-kube-api-access-gzn5t\") pod \"kserve-controller-manager-d6f4b65b8-cfkpl\" (UID: \"787d4d5b-c973-4a3c-956b-c53163ef2b3a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" Jul 23 01:34:41.991319 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:41.991214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/787d4d5b-c973-4a3c-956b-c53163ef2b3a-cert\") pod \"kserve-controller-manager-d6f4b65b8-cfkpl\" (UID: \"787d4d5b-c973-4a3c-956b-c53163ef2b3a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" Jul 23 01:34:41.991319 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:34:41.991229 2572 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 23 01:34:41.991319 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:34:41.991303 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c96e9f7-295f-401d-8459-b336023730c3-cert podName:8c96e9f7-295f-401d-8459-b336023730c3 nodeName:}" failed. No retries permitted until 2026-07-23 01:34:42.491280011 +0000 UTC m=+395.036229170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8c96e9f7-295f-401d-8459-b336023730c3-cert") pod "llmisvc-controller-manager-777c6bd7d5-258ws" (UID: "8c96e9f7-295f-401d-8459-b336023730c3") : secret "llmisvc-webhook-server-cert" not found Jul 23 01:34:41.991532 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:34:41.991352 2572 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 23 01:34:41.991532 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:34:41.991403 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/787d4d5b-c973-4a3c-956b-c53163ef2b3a-cert podName:787d4d5b-c973-4a3c-956b-c53163ef2b3a nodeName:}" failed. No retries permitted until 2026-07-23 01:34:42.491385347 +0000 UTC m=+395.036334511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/787d4d5b-c973-4a3c-956b-c53163ef2b3a-cert") pod "kserve-controller-manager-d6f4b65b8-cfkpl" (UID: "787d4d5b-c973-4a3c-956b-c53163ef2b3a") : secret "kserve-webhook-server-cert" not found Jul 23 01:34:42.018267 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:42.018234 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzn5t\" (UniqueName: \"kubernetes.io/projected/787d4d5b-c973-4a3c-956b-c53163ef2b3a-kube-api-access-gzn5t\") pod \"kserve-controller-manager-d6f4b65b8-cfkpl\" (UID: \"787d4d5b-c973-4a3c-956b-c53163ef2b3a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" Jul 23 01:34:42.020768 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:42.020740 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc6wk\" (UniqueName: \"kubernetes.io/projected/8c96e9f7-295f-401d-8459-b336023730c3-kube-api-access-wc6wk\") pod \"llmisvc-controller-manager-777c6bd7d5-258ws\" (UID: \"8c96e9f7-295f-401d-8459-b336023730c3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" Jul 23 01:34:42.495795 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:42.495760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c96e9f7-295f-401d-8459-b336023730c3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-258ws\" (UID: \"8c96e9f7-295f-401d-8459-b336023730c3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" Jul 23 01:34:42.495950 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:42.495847 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/787d4d5b-c973-4a3c-956b-c53163ef2b3a-cert\") pod \"kserve-controller-manager-d6f4b65b8-cfkpl\" (UID: \"787d4d5b-c973-4a3c-956b-c53163ef2b3a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" Jul 23 01:34:42.498248 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:42.498217 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/787d4d5b-c973-4a3c-956b-c53163ef2b3a-cert\") pod \"kserve-controller-manager-d6f4b65b8-cfkpl\" (UID: \"787d4d5b-c973-4a3c-956b-c53163ef2b3a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" Jul 23 01:34:42.498370 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:42.498220 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c96e9f7-295f-401d-8459-b336023730c3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-258ws\" (UID: \"8c96e9f7-295f-401d-8459-b336023730c3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" Jul 23 01:34:42.694762 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:42.694733 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" Jul 23 01:34:42.705370 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:42.705345 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" Jul 23 01:34:42.849011 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:42.848929 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl"] Jul 23 01:34:42.851407 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:34:42.851378 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787d4d5b_c973_4a3c_956b_c53163ef2b3a.slice/crio-d5e82d984d3b3951557a9ea5eac74561085c7ed086848d338d8fd4a98d572edd WatchSource:0}: Error finding container d5e82d984d3b3951557a9ea5eac74561085c7ed086848d338d8fd4a98d572edd: Status 404 returned error can't find the container with id d5e82d984d3b3951557a9ea5eac74561085c7ed086848d338d8fd4a98d572edd Jul 23 01:34:42.870579 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:42.870557 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws"] Jul 23 01:34:42.873619 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:34:42.873591 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8c96e9f7_295f_401d_8459_b336023730c3.slice/crio-f09deba009fa17e7de416fd0b40df6f0a51e94e95fdacbeed093a4a51f216e40 WatchSource:0}: Error finding container f09deba009fa17e7de416fd0b40df6f0a51e94e95fdacbeed093a4a51f216e40: Status 404 returned error can't find the container with id f09deba009fa17e7de416fd0b40df6f0a51e94e95fdacbeed093a4a51f216e40 Jul 23 01:34:43.678694 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:43.678577 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" event={"ID":"8c96e9f7-295f-401d-8459-b336023730c3","Type":"ContainerStarted","Data":"f09deba009fa17e7de416fd0b40df6f0a51e94e95fdacbeed093a4a51f216e40"} Jul 23 01:34:43.681430 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:43.681372 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" event={"ID":"787d4d5b-c973-4a3c-956b-c53163ef2b3a","Type":"ContainerStarted","Data":"d5e82d984d3b3951557a9ea5eac74561085c7ed086848d338d8fd4a98d572edd"} Jul 23 01:34:45.706143 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:45.706111 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" event={"ID":"787d4d5b-c973-4a3c-956b-c53163ef2b3a","Type":"ContainerStarted","Data":"21b9be5e0a3eb44234e68af634bd44271e7b51bd4e541f4cad9d7a1a4e6bcbac"} Jul 23 01:34:45.706589 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:45.706195 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" Jul 23 01:34:45.725234 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:45.725180 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" podStartSLOduration=2.42276238 podStartE2EDuration="4.725159001s" totalImagesPullingTime="2.302396621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:34:45.722622461 +0000 UTC m=+398.267571639" watchObservedRunningTime="2026-07-23 01:34:45.725159001 +0000 UTC m=+398.270108176" Jul 23 01:34:46.714815 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:46.714773 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" event={"ID":"8c96e9f7-295f-401d-8459-b336023730c3","Type":"ContainerStarted","Data":"27179a41dd9e1b0bc99f44defd6f262870af1131a4b64f982b808fe66f131f72"} Jul 23 01:34:46.715332 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:46.714845 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" Jul 23 01:34:46.730937 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:46.730891 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" podStartSLOduration=2.556208786 podStartE2EDuration="5.730875055s" totalImagesPullingTime="3.174666269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:34:46.72969484 +0000 UTC m=+399.274644031" watchObservedRunningTime="2026-07-23 01:34:46.730875055 +0000 UTC m=+399.275824225" Jul 23 01:34:48.646085 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:48.646051 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-xkv9t" Jul 23 01:34:48.649177 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:34:48.649152 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-tf2sm" Jul 23 01:35:16.719795 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:16.719759 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cfkpl" Jul 23 01:35:17.722414 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:17.722374 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-258ws" Jul 23 01:35:43.761565 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.761473 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99"] Jul 23 01:35:43.765302 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.765277 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:43.767470 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.767448 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 01:35:43.767567 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.767486 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 01:35:43.768203 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.768189 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-x2gdz\"" Jul 23 01:35:43.771760 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.771736 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99"] Jul 23 01:35:43.896158 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.896116 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79rlp\" (UniqueName: \"kubernetes.io/projected/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-kube-api-access-79rlp\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99\" (UID: \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:43.896330 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.896170 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99\" (UID: \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:43.896330 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.896218 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99\" (UID: \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:43.997166 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.997135 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99\" (UID: \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:43.997166 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.997170 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99\" (UID: \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:43.997400 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.997233 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79rlp\" (UniqueName: \"kubernetes.io/projected/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-kube-api-access-79rlp\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99\" (UID: \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:43.997528 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.997507 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99\" (UID: \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:43.997571 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:43.997544 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99\" (UID: \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:44.006775 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.006740 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79rlp\" (UniqueName: \"kubernetes.io/projected/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-kube-api-access-79rlp\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99\" (UID: \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:44.075770 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.075678 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:44.206155 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.206123 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99"] Jul 23 01:35:44.209529 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:35:44.209497 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ee4acd_7842_45dc_8dcf_0a266c5e1fd3.slice/crio-416205b137123a408831f32cb4c08aa9395bce4b3048a2b3ff89a9c94d36bbe1 WatchSource:0}: Error finding container 416205b137123a408831f32cb4c08aa9395bce4b3048a2b3ff89a9c94d36bbe1: Status 404 returned error can't find the container with id 416205b137123a408831f32cb4c08aa9395bce4b3048a2b3ff89a9c94d36bbe1 Jul 23 01:35:44.357739 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.357703 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp"] Jul 23 01:35:44.361029 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.361006 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:44.368334 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.368310 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp"] Jul 23 01:35:44.400525 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.400492 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7p6\" (UniqueName: \"kubernetes.io/projected/bea38df4-c56c-44f3-959f-9d948eeb67aa-kube-api-access-mq7p6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp\" (UID: \"bea38df4-c56c-44f3-959f-9d948eeb67aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:44.400704 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.400546 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bea38df4-c56c-44f3-959f-9d948eeb67aa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp\" (UID: \"bea38df4-c56c-44f3-959f-9d948eeb67aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:44.400704 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.400614 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bea38df4-c56c-44f3-959f-9d948eeb67aa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp\" (UID: \"bea38df4-c56c-44f3-959f-9d948eeb67aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:44.501560 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.501451 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bea38df4-c56c-44f3-959f-9d948eeb67aa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp\" (UID: \"bea38df4-c56c-44f3-959f-9d948eeb67aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:44.501781 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.501598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bea38df4-c56c-44f3-959f-9d948eeb67aa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp\" (UID: \"bea38df4-c56c-44f3-959f-9d948eeb67aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:44.501781 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.501695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7p6\" (UniqueName: \"kubernetes.io/projected/bea38df4-c56c-44f3-959f-9d948eeb67aa-kube-api-access-mq7p6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp\" (UID: \"bea38df4-c56c-44f3-959f-9d948eeb67aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:44.501911 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.501878 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bea38df4-c56c-44f3-959f-9d948eeb67aa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp\" (UID: \"bea38df4-c56c-44f3-959f-9d948eeb67aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:44.501958 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.501908 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bea38df4-c56c-44f3-959f-9d948eeb67aa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp\" (UID: \"bea38df4-c56c-44f3-959f-9d948eeb67aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:44.510473 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.510454 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7p6\" (UniqueName: \"kubernetes.io/projected/bea38df4-c56c-44f3-959f-9d948eeb67aa-kube-api-access-mq7p6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp\" (UID: \"bea38df4-c56c-44f3-959f-9d948eeb67aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:44.671398 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.671313 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:44.800330 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.800298 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp"] Jul 23 01:35:44.802045 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:35:44.802017 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea38df4_c56c_44f3_959f_9d948eeb67aa.slice/crio-0f29781c3815cf86555fa8a411840c71e0704b675c96b1048d590ef809aa0029 WatchSource:0}: Error finding container 0f29781c3815cf86555fa8a411840c71e0704b675c96b1048d590ef809aa0029: Status 404 returned error can't find the container with id 0f29781c3815cf86555fa8a411840c71e0704b675c96b1048d590ef809aa0029 Jul 23 01:35:44.960455 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.960368 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf"] Jul 23 01:35:44.963658 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.963613 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:44.971380 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.971356 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf"] Jul 23 01:35:44.991879 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.991842 2572 generic.go:356] "Generic (PLEG): container finished" podID="e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" containerID="6dc4b16b1b471bd810f228a2ad766436dc0f977dd03827320e444db157b7fcb5" exitCode=0 Jul 23 01:35:44.992039 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.991917 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" event={"ID":"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3","Type":"ContainerDied","Data":"6dc4b16b1b471bd810f228a2ad766436dc0f977dd03827320e444db157b7fcb5"} Jul 23 01:35:44.992039 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.991988 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" event={"ID":"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3","Type":"ContainerStarted","Data":"416205b137123a408831f32cb4c08aa9395bce4b3048a2b3ff89a9c94d36bbe1"} Jul 23 01:35:44.993665 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.993627 2572 generic.go:356] "Generic (PLEG): container finished" podID="bea38df4-c56c-44f3-959f-9d948eeb67aa" containerID="597b6b89cc88d77938678e738803734ecff49b91f001e04b86f10b2059729054" exitCode=0 Jul 23 01:35:44.993778 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.993684 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" event={"ID":"bea38df4-c56c-44f3-959f-9d948eeb67aa","Type":"ContainerDied","Data":"597b6b89cc88d77938678e738803734ecff49b91f001e04b86f10b2059729054"} Jul 23 01:35:44.993778 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:44.993709 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" event={"ID":"bea38df4-c56c-44f3-959f-9d948eeb67aa","Type":"ContainerStarted","Data":"0f29781c3815cf86555fa8a411840c71e0704b675c96b1048d590ef809aa0029"} Jul 23 01:35:45.005530 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.005503 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adfb6570-f16f-4985-b6db-2a07c0aab0e9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf\" (UID: \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:45.005654 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.005549 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd65q\" (UniqueName: \"kubernetes.io/projected/adfb6570-f16f-4985-b6db-2a07c0aab0e9-kube-api-access-vd65q\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf\" (UID: \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:45.005771 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.005748 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adfb6570-f16f-4985-b6db-2a07c0aab0e9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf\" (UID: \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:45.106564 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.106514 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd65q\" (UniqueName: \"kubernetes.io/projected/adfb6570-f16f-4985-b6db-2a07c0aab0e9-kube-api-access-vd65q\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf\" (UID: \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:45.106797 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.106699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adfb6570-f16f-4985-b6db-2a07c0aab0e9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf\" (UID: \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:45.106797 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.106739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adfb6570-f16f-4985-b6db-2a07c0aab0e9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf\" (UID: \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:45.107115 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.107092 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adfb6570-f16f-4985-b6db-2a07c0aab0e9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf\" (UID: \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:45.107184 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.107132 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adfb6570-f16f-4985-b6db-2a07c0aab0e9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf\" (UID: \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:45.115241 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.115215 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd65q\" (UniqueName: \"kubernetes.io/projected/adfb6570-f16f-4985-b6db-2a07c0aab0e9-kube-api-access-vd65q\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf\" (UID: \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:45.282715 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.282596 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:45.412772 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.412741 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf"] Jul 23 01:35:45.414532 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:35:45.414505 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfb6570_f16f_4985_b6db_2a07c0aab0e9.slice/crio-bdf82552e2c0ea2d4556e58d64a887da751d188645c0abf63a2922a1af636832 WatchSource:0}: Error finding container bdf82552e2c0ea2d4556e58d64a887da751d188645c0abf63a2922a1af636832: Status 404 returned error can't find the container with id bdf82552e2c0ea2d4556e58d64a887da751d188645c0abf63a2922a1af636832 Jul 23 01:35:45.564644 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.564568 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm"] Jul 23 01:35:45.567858 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.567842 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:45.574664 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.574618 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm"] Jul 23 01:35:45.610531 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.610502 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm\" (UID: \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:45.610720 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.610547 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv2jt\" (UniqueName: \"kubernetes.io/projected/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-kube-api-access-qv2jt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm\" (UID: \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:45.610720 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.610688 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm\" (UID: \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:45.711873 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.711837 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm\" (UID: \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:45.712048 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.711925 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm\" (UID: \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:45.712048 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.711950 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv2jt\" (UniqueName: \"kubernetes.io/projected/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-kube-api-access-qv2jt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm\" (UID: \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:45.712188 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.712167 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm\" (UID: \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:45.712287 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.712264 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm\" (UID: \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:45.720219 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.720198 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv2jt\" (UniqueName: \"kubernetes.io/projected/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-kube-api-access-qv2jt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm\" (UID: \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:45.882492 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:45.882458 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:46.000468 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:46.000436 2572 generic.go:356] "Generic (PLEG): container finished" podID="e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" containerID="d8149b4cfeed2ee24cb2cde77caa7460586010bc3dff46093e5705a5c1fee9ab" exitCode=0 Jul 23 01:35:46.000691 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:46.000513 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" event={"ID":"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3","Type":"ContainerDied","Data":"d8149b4cfeed2ee24cb2cde77caa7460586010bc3dff46093e5705a5c1fee9ab"} Jul 23 01:35:46.002463 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:46.002440 2572 generic.go:356] "Generic (PLEG): container finished" podID="bea38df4-c56c-44f3-959f-9d948eeb67aa" containerID="c2db9d5a16444acb481d8cf5e1695203809726f8f28aa08242c86cdae0d6e9c2" exitCode=0 Jul 23 01:35:46.002549 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:46.002517 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" event={"ID":"bea38df4-c56c-44f3-959f-9d948eeb67aa","Type":"ContainerDied","Data":"c2db9d5a16444acb481d8cf5e1695203809726f8f28aa08242c86cdae0d6e9c2"} Jul 23 01:35:46.004112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:46.004088 2572 generic.go:356] "Generic (PLEG): container finished" podID="adfb6570-f16f-4985-b6db-2a07c0aab0e9" containerID="29f3e5a6d4f7affd35f34aaa54662b9eb442f1b0af76b5cda04e28c24551f58d" exitCode=0 Jul 23 01:35:46.004182 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:46.004108 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" event={"ID":"adfb6570-f16f-4985-b6db-2a07c0aab0e9","Type":"ContainerDied","Data":"29f3e5a6d4f7affd35f34aaa54662b9eb442f1b0af76b5cda04e28c24551f58d"} Jul 23 01:35:46.004182 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:46.004149 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" event={"ID":"adfb6570-f16f-4985-b6db-2a07c0aab0e9","Type":"ContainerStarted","Data":"bdf82552e2c0ea2d4556e58d64a887da751d188645c0abf63a2922a1af636832"} Jul 23 01:35:46.017797 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:46.017770 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm"] Jul 23 01:35:46.020438 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:35:46.020413 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806a17ab_62e4_4ce5_b109_f51c7f50f4f9.slice/crio-cdffbf69aae1aac58e311e5239255185e6b88f7a1579f98670d189268bec8ed0 WatchSource:0}: Error finding container cdffbf69aae1aac58e311e5239255185e6b88f7a1579f98670d189268bec8ed0: Status 404 returned error can't find the container with id cdffbf69aae1aac58e311e5239255185e6b88f7a1579f98670d189268bec8ed0 Jul 23 01:35:47.014112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:47.014083 2572 generic.go:356] "Generic (PLEG): container finished" podID="bea38df4-c56c-44f3-959f-9d948eeb67aa" containerID="552755465af746a28e43cd0ac85087ff2101a48a0e28c059b0b872395d8e27cd" exitCode=0 Jul 23 01:35:47.014500 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:47.014165 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" event={"ID":"bea38df4-c56c-44f3-959f-9d948eeb67aa","Type":"ContainerDied","Data":"552755465af746a28e43cd0ac85087ff2101a48a0e28c059b0b872395d8e27cd"} Jul 23 01:35:47.015752 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:47.015731 2572 generic.go:356] "Generic (PLEG): container finished" podID="adfb6570-f16f-4985-b6db-2a07c0aab0e9" containerID="7856df19ae9b9910b2efd1373b0e780f31bc6a1d7df4adfebf2c50744b634091" exitCode=0 Jul 23 01:35:47.015850 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:47.015807 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" event={"ID":"adfb6570-f16f-4985-b6db-2a07c0aab0e9","Type":"ContainerDied","Data":"7856df19ae9b9910b2efd1373b0e780f31bc6a1d7df4adfebf2c50744b634091"} Jul 23 01:35:47.017216 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:47.017192 2572 generic.go:356] "Generic (PLEG): container finished" podID="806a17ab-62e4-4ce5-b109-f51c7f50f4f9" containerID="8470e71c4b936e1aaa7f46ef9c89f71387ac967e2c459c2ed1d198858b7d127e" exitCode=0 Jul 23 01:35:47.017454 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:47.017219 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" event={"ID":"806a17ab-62e4-4ce5-b109-f51c7f50f4f9","Type":"ContainerDied","Data":"8470e71c4b936e1aaa7f46ef9c89f71387ac967e2c459c2ed1d198858b7d127e"} Jul 23 01:35:47.017454 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:47.017260 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" event={"ID":"806a17ab-62e4-4ce5-b109-f51c7f50f4f9","Type":"ContainerStarted","Data":"cdffbf69aae1aac58e311e5239255185e6b88f7a1579f98670d189268bec8ed0"} Jul 23 01:35:47.019590 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:47.019567 2572 generic.go:356] "Generic (PLEG): container finished" podID="e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" containerID="17080b5b10e11b529b6e3e031810a2930e20fd0d6cf217981773e56ec730fe3d" exitCode=0 Jul 23 01:35:47.019685 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:47.019669 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" event={"ID":"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3","Type":"ContainerDied","Data":"17080b5b10e11b529b6e3e031810a2930e20fd0d6cf217981773e56ec730fe3d"} Jul 23 01:35:48.027507 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.027474 2572 generic.go:356] "Generic (PLEG): container finished" podID="adfb6570-f16f-4985-b6db-2a07c0aab0e9" containerID="be83a5d0924e478cc3db664c0127f079a053ddba2bad54817c5f53223b4c25ce" exitCode=0 Jul 23 01:35:48.028008 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.027575 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" event={"ID":"adfb6570-f16f-4985-b6db-2a07c0aab0e9","Type":"ContainerDied","Data":"be83a5d0924e478cc3db664c0127f079a053ddba2bad54817c5f53223b4c25ce"} Jul 23 01:35:48.029318 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.029294 2572 generic.go:356] "Generic (PLEG): container finished" podID="806a17ab-62e4-4ce5-b109-f51c7f50f4f9" containerID="8592f30fb41c131462d79d386b57df86ba9bd8255a33ee1d548abdfe21fc260a" exitCode=0 Jul 23 01:35:48.029457 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.029382 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" event={"ID":"806a17ab-62e4-4ce5-b109-f51c7f50f4f9","Type":"ContainerDied","Data":"8592f30fb41c131462d79d386b57df86ba9bd8255a33ee1d548abdfe21fc260a"} Jul 23 01:35:48.179362 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.179296 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:48.236218 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.236192 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-util\") pod \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\" (UID: \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\") " Jul 23 01:35:48.236394 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.236291 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-bundle\") pod \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\" (UID: \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\") " Jul 23 01:35:48.236394 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.236329 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79rlp\" (UniqueName: \"kubernetes.io/projected/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-kube-api-access-79rlp\") pod \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\" (UID: \"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3\") " Jul 23 01:35:48.236920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.236889 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-bundle" (OuterVolumeSpecName: "bundle") pod "e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" (UID: "e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:35:48.238549 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.238521 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-kube-api-access-79rlp" (OuterVolumeSpecName: "kube-api-access-79rlp") pod "e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" (UID: "e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3"). InnerVolumeSpecName "kube-api-access-79rlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:35:48.241749 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.241724 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-util" (OuterVolumeSpecName: "util") pod "e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" (UID: "e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:35:48.248852 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.248833 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:48.337030 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.336998 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bea38df4-c56c-44f3-959f-9d948eeb67aa-util\") pod \"bea38df4-c56c-44f3-959f-9d948eeb67aa\" (UID: \"bea38df4-c56c-44f3-959f-9d948eeb67aa\") " Jul 23 01:35:48.337227 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.337053 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bea38df4-c56c-44f3-959f-9d948eeb67aa-bundle\") pod \"bea38df4-c56c-44f3-959f-9d948eeb67aa\" (UID: \"bea38df4-c56c-44f3-959f-9d948eeb67aa\") " Jul 23 01:35:48.337227 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.337090 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq7p6\" (UniqueName: \"kubernetes.io/projected/bea38df4-c56c-44f3-959f-9d948eeb67aa-kube-api-access-mq7p6\") pod \"bea38df4-c56c-44f3-959f-9d948eeb67aa\" (UID: \"bea38df4-c56c-44f3-959f-9d948eeb67aa\") " Jul 23 01:35:48.337347 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.337243 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-bundle\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:35:48.337347 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.337261 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79rlp\" (UniqueName: \"kubernetes.io/projected/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-kube-api-access-79rlp\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:35:48.337347 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.337275 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3-util\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:35:48.337626 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.337600 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea38df4-c56c-44f3-959f-9d948eeb67aa-bundle" (OuterVolumeSpecName: "bundle") pod "bea38df4-c56c-44f3-959f-9d948eeb67aa" (UID: "bea38df4-c56c-44f3-959f-9d948eeb67aa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:35:48.339175 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.339152 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea38df4-c56c-44f3-959f-9d948eeb67aa-kube-api-access-mq7p6" (OuterVolumeSpecName: "kube-api-access-mq7p6") pod "bea38df4-c56c-44f3-959f-9d948eeb67aa" (UID: "bea38df4-c56c-44f3-959f-9d948eeb67aa"). InnerVolumeSpecName "kube-api-access-mq7p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:35:48.342428 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.342394 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea38df4-c56c-44f3-959f-9d948eeb67aa-util" (OuterVolumeSpecName: "util") pod "bea38df4-c56c-44f3-959f-9d948eeb67aa" (UID: "bea38df4-c56c-44f3-959f-9d948eeb67aa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:35:48.438353 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.438276 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bea38df4-c56c-44f3-959f-9d948eeb67aa-bundle\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:35:48.438353 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.438308 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq7p6\" (UniqueName: \"kubernetes.io/projected/bea38df4-c56c-44f3-959f-9d948eeb67aa-kube-api-access-mq7p6\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:35:48.438353 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:48.438317 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bea38df4-c56c-44f3-959f-9d948eeb67aa-util\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:35:49.036236 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.036206 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" Jul 23 01:35:49.036236 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.036228 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99" event={"ID":"e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3","Type":"ContainerDied","Data":"416205b137123a408831f32cb4c08aa9395bce4b3048a2b3ff89a9c94d36bbe1"} Jul 23 01:35:49.036742 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.036279 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="416205b137123a408831f32cb4c08aa9395bce4b3048a2b3ff89a9c94d36bbe1" Jul 23 01:35:49.038106 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.038085 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" Jul 23 01:35:49.038243 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.038108 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp" event={"ID":"bea38df4-c56c-44f3-959f-9d948eeb67aa","Type":"ContainerDied","Data":"0f29781c3815cf86555fa8a411840c71e0704b675c96b1048d590ef809aa0029"} Jul 23 01:35:49.038243 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.038149 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f29781c3815cf86555fa8a411840c71e0704b675c96b1048d590ef809aa0029" Jul 23 01:35:49.040250 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.040225 2572 generic.go:356] "Generic (PLEG): container finished" podID="806a17ab-62e4-4ce5-b109-f51c7f50f4f9" containerID="4dead63725bf75b995295ea71085edeebe3c682702034cd7b4e6b7373a78d6b8" exitCode=0 Jul 23 01:35:49.040344 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.040283 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" event={"ID":"806a17ab-62e4-4ce5-b109-f51c7f50f4f9","Type":"ContainerDied","Data":"4dead63725bf75b995295ea71085edeebe3c682702034cd7b4e6b7373a78d6b8"} Jul 23 01:35:49.179689 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.179649 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:49.244367 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.244331 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adfb6570-f16f-4985-b6db-2a07c0aab0e9-util\") pod \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\" (UID: \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\") " Jul 23 01:35:49.244554 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.244377 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd65q\" (UniqueName: \"kubernetes.io/projected/adfb6570-f16f-4985-b6db-2a07c0aab0e9-kube-api-access-vd65q\") pod \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\" (UID: \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\") " Jul 23 01:35:49.244554 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.244480 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adfb6570-f16f-4985-b6db-2a07c0aab0e9-bundle\") pod \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\" (UID: \"adfb6570-f16f-4985-b6db-2a07c0aab0e9\") " Jul 23 01:35:49.244991 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.244961 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfb6570-f16f-4985-b6db-2a07c0aab0e9-bundle" (OuterVolumeSpecName: "bundle") pod "adfb6570-f16f-4985-b6db-2a07c0aab0e9" (UID: "adfb6570-f16f-4985-b6db-2a07c0aab0e9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:35:49.246482 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.246458 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfb6570-f16f-4985-b6db-2a07c0aab0e9-kube-api-access-vd65q" (OuterVolumeSpecName: "kube-api-access-vd65q") pod "adfb6570-f16f-4985-b6db-2a07c0aab0e9" (UID: "adfb6570-f16f-4985-b6db-2a07c0aab0e9"). InnerVolumeSpecName "kube-api-access-vd65q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:35:49.249769 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.249747 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfb6570-f16f-4985-b6db-2a07c0aab0e9-util" (OuterVolumeSpecName: "util") pod "adfb6570-f16f-4985-b6db-2a07c0aab0e9" (UID: "adfb6570-f16f-4985-b6db-2a07c0aab0e9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:35:49.346083 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.346056 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd65q\" (UniqueName: \"kubernetes.io/projected/adfb6570-f16f-4985-b6db-2a07c0aab0e9-kube-api-access-vd65q\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:35:49.346227 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.346085 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adfb6570-f16f-4985-b6db-2a07c0aab0e9-bundle\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:35:49.346227 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:49.346097 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adfb6570-f16f-4985-b6db-2a07c0aab0e9-util\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:35:50.046708 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.046679 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" Jul 23 01:35:50.047096 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.046677 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf" event={"ID":"adfb6570-f16f-4985-b6db-2a07c0aab0e9","Type":"ContainerDied","Data":"bdf82552e2c0ea2d4556e58d64a887da751d188645c0abf63a2922a1af636832"} Jul 23 01:35:50.047096 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.046817 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdf82552e2c0ea2d4556e58d64a887da751d188645c0abf63a2922a1af636832" Jul 23 01:35:50.178344 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.178321 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:50.255628 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.255540 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-util\") pod \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\" (UID: \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\") " Jul 23 01:35:50.255628 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.255593 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-bundle\") pod \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\" (UID: \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\") " Jul 23 01:35:50.255840 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.255668 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv2jt\" (UniqueName: \"kubernetes.io/projected/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-kube-api-access-qv2jt\") pod \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\" (UID: \"806a17ab-62e4-4ce5-b109-f51c7f50f4f9\") " Jul 23 01:35:50.256164 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.256132 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-bundle" (OuterVolumeSpecName: "bundle") pod "806a17ab-62e4-4ce5-b109-f51c7f50f4f9" (UID: "806a17ab-62e4-4ce5-b109-f51c7f50f4f9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:35:50.257817 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.257789 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-kube-api-access-qv2jt" (OuterVolumeSpecName: "kube-api-access-qv2jt") pod "806a17ab-62e4-4ce5-b109-f51c7f50f4f9" (UID: "806a17ab-62e4-4ce5-b109-f51c7f50f4f9"). InnerVolumeSpecName "kube-api-access-qv2jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:35:50.260853 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.260820 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-util" (OuterVolumeSpecName: "util") pod "806a17ab-62e4-4ce5-b109-f51c7f50f4f9" (UID: "806a17ab-62e4-4ce5-b109-f51c7f50f4f9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:35:50.356787 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.356749 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv2jt\" (UniqueName: \"kubernetes.io/projected/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-kube-api-access-qv2jt\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:35:50.356787 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.356781 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-util\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:35:50.356787 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:50.356790 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/806a17ab-62e4-4ce5-b109-f51c7f50f4f9-bundle\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:35:51.052889 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:51.052850 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" event={"ID":"806a17ab-62e4-4ce5-b109-f51c7f50f4f9","Type":"ContainerDied","Data":"cdffbf69aae1aac58e311e5239255185e6b88f7a1579f98670d189268bec8ed0"} Jul 23 01:35:51.052889 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:51.052878 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm" Jul 23 01:35:51.053348 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:51.052909 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdffbf69aae1aac58e311e5239255185e6b88f7a1579f98670d189268bec8ed0" Jul 23 01:35:57.877689 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.877652 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bfc6b5b57-5mlpw"] Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.877958 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" containerName="extract" Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.877968 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" containerName="extract" Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.877988 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" containerName="util" Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.877993 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" containerName="util" Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878000 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bea38df4-c56c-44f3-959f-9d948eeb67aa" containerName="extract" Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878005 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bea38df4-c56c-44f3-959f-9d948eeb67aa" containerName="extract" Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878011 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="806a17ab-62e4-4ce5-b109-f51c7f50f4f9" containerName="pull" Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878016 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="806a17ab-62e4-4ce5-b109-f51c7f50f4f9" containerName="pull" Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878025 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="806a17ab-62e4-4ce5-b109-f51c7f50f4f9" containerName="util" Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878030 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="806a17ab-62e4-4ce5-b109-f51c7f50f4f9" containerName="util" Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878036 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="806a17ab-62e4-4ce5-b109-f51c7f50f4f9" containerName="extract" Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878042 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="806a17ab-62e4-4ce5-b109-f51c7f50f4f9" containerName="extract" Jul 23 01:35:57.878081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878048 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bea38df4-c56c-44f3-959f-9d948eeb67aa" containerName="util" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878165 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bea38df4-c56c-44f3-959f-9d948eeb67aa" containerName="util" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878174 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" containerName="pull" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878179 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" containerName="pull" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878186 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfb6570-f16f-4985-b6db-2a07c0aab0e9" containerName="pull" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878190 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfb6570-f16f-4985-b6db-2a07c0aab0e9" containerName="pull" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878196 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfb6570-f16f-4985-b6db-2a07c0aab0e9" containerName="extract" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878201 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfb6570-f16f-4985-b6db-2a07c0aab0e9" containerName="extract" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878209 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfb6570-f16f-4985-b6db-2a07c0aab0e9" containerName="util" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878214 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfb6570-f16f-4985-b6db-2a07c0aab0e9" containerName="util" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878221 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bea38df4-c56c-44f3-959f-9d948eeb67aa" containerName="pull" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878225 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bea38df4-c56c-44f3-959f-9d948eeb67aa" containerName="pull" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878295 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfb6570-f16f-4985-b6db-2a07c0aab0e9" containerName="extract" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878307 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bea38df4-c56c-44f3-959f-9d948eeb67aa" containerName="extract" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878313 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="806a17ab-62e4-4ce5-b109-f51c7f50f4f9" containerName="extract" Jul 23 01:35:57.878465 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.878319 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3" containerName="extract" Jul 23 01:35:57.881442 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.881425 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:57.891839 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:57.891811 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bfc6b5b57-5mlpw"] Jul 23 01:35:58.029112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.029080 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kztvc\" (UniqueName: \"kubernetes.io/projected/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-kube-api-access-kztvc\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.029112 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.029117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-console-config\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.029340 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.029150 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-console-serving-cert\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.029340 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.029225 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-service-ca\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.029340 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.029269 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-oauth-serving-cert\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.029340 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.029303 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-trusted-ca-bundle\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.029479 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.029342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-console-oauth-config\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.130074 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.129970 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-trusted-ca-bundle\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.130074 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.130029 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-console-oauth-config\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.130074 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.130050 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kztvc\" (UniqueName: \"kubernetes.io/projected/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-kube-api-access-kztvc\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.130383 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.130081 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-console-config\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.130383 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.130127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-console-serving-cert\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.130383 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.130158 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-service-ca\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.130383 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.130373 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-oauth-serving-cert\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.130941 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.130915 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-service-ca\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.131059 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.130915 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-console-config\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.131131 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.131073 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-oauth-serving-cert\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.131442 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.131417 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-trusted-ca-bundle\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.132781 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.132762 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-console-oauth-config\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.132874 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.132801 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-console-serving-cert\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.139585 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.139567 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kztvc\" (UniqueName: \"kubernetes.io/projected/ecf4c727-68a7-473e-a8a0-83c04ec27eb0-kube-api-access-kztvc\") pod \"console-7bfc6b5b57-5mlpw\" (UID: \"ecf4c727-68a7-473e-a8a0-83c04ec27eb0\") " pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.191382 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.191343 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:35:58.324894 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:58.324867 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bfc6b5b57-5mlpw"] Jul 23 01:35:58.326509 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:35:58.326482 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf4c727_68a7_473e_a8a0_83c04ec27eb0.slice/crio-661f88493680b93bd84be7a3e35b2e15733f8c60fc2450a55753a07b8359d0a2 WatchSource:0}: Error finding container 661f88493680b93bd84be7a3e35b2e15733f8c60fc2450a55753a07b8359d0a2: Status 404 returned error can't find the container with id 661f88493680b93bd84be7a3e35b2e15733f8c60fc2450a55753a07b8359d0a2 Jul 23 01:35:59.093905 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:59.093866 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bfc6b5b57-5mlpw" event={"ID":"ecf4c727-68a7-473e-a8a0-83c04ec27eb0","Type":"ContainerStarted","Data":"2feffd09bc0fb274a8ba3e32a3e7a32fa715368128ea9db0e381cb328b9bdf62"} Jul 23 01:35:59.094391 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:59.093922 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bfc6b5b57-5mlpw" event={"ID":"ecf4c727-68a7-473e-a8a0-83c04ec27eb0","Type":"ContainerStarted","Data":"661f88493680b93bd84be7a3e35b2e15733f8c60fc2450a55753a07b8359d0a2"} Jul 23 01:35:59.109759 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:35:59.109711 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bfc6b5b57-5mlpw" podStartSLOduration=2.109697427 podStartE2EDuration="2.109697427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:35:59.109056676 +0000 UTC m=+471.654005847" watchObservedRunningTime="2026-07-23 01:35:59.109697427 +0000 UTC m=+471.654646600" Jul 23 01:36:07.329743 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:07.329703 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw"] Jul 23 01:36:07.334580 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:07.334563 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" Jul 23 01:36:07.337102 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:07.337079 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7tl77\"" Jul 23 01:36:07.337395 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:07.337381 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 01:36:07.338014 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:07.337996 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 01:36:07.343998 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:07.343968 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw"] Jul 23 01:36:07.519821 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:07.519791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr9kh\" (UniqueName: \"kubernetes.io/projected/2e1b5b8c-f233-42d3-881d-b14da3db631e-kube-api-access-qr9kh\") pod \"limitador-operator-controller-manager-859d7b78bd-npbfw\" (UID: \"2e1b5b8c-f233-42d3-881d-b14da3db631e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" Jul 23 01:36:07.620658 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:07.620606 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr9kh\" (UniqueName: \"kubernetes.io/projected/2e1b5b8c-f233-42d3-881d-b14da3db631e-kube-api-access-qr9kh\") pod \"limitador-operator-controller-manager-859d7b78bd-npbfw\" (UID: \"2e1b5b8c-f233-42d3-881d-b14da3db631e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" Jul 23 01:36:07.629767 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:07.629734 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr9kh\" (UniqueName: \"kubernetes.io/projected/2e1b5b8c-f233-42d3-881d-b14da3db631e-kube-api-access-qr9kh\") pod \"limitador-operator-controller-manager-859d7b78bd-npbfw\" (UID: \"2e1b5b8c-f233-42d3-881d-b14da3db631e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" Jul 23 01:36:07.646534 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:07.646508 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" Jul 23 01:36:07.780402 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:07.780368 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw"] Jul 23 01:36:07.784929 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:36:07.784895 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1b5b8c_f233_42d3_881d_b14da3db631e.slice/crio-61cf429f903df830449f1ae05a8d0465a28612f6aa0d358f2af82c16cf151503 WatchSource:0}: Error finding container 61cf429f903df830449f1ae05a8d0465a28612f6aa0d358f2af82c16cf151503: Status 404 returned error can't find the container with id 61cf429f903df830449f1ae05a8d0465a28612f6aa0d358f2af82c16cf151503 Jul 23 01:36:08.139506 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:08.139470 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" event={"ID":"2e1b5b8c-f233-42d3-881d-b14da3db631e","Type":"ContainerStarted","Data":"61cf429f903df830449f1ae05a8d0465a28612f6aa0d358f2af82c16cf151503"} Jul 23 01:36:08.191619 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:08.191577 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:36:08.191799 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:08.191681 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:36:08.197012 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:08.196991 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:36:09.148281 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:09.148250 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bfc6b5b57-5mlpw" Jul 23 01:36:09.197729 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:09.197694 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f85c8bbb6-xckr4"] Jul 23 01:36:10.150044 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:10.149994 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" event={"ID":"2e1b5b8c-f233-42d3-881d-b14da3db631e","Type":"ContainerStarted","Data":"9393515a03e27872b6f7c5ace96c8f480e84726242ecf5dc834e3600e80807f9"} Jul 23 01:36:10.165370 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:10.165323 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" podStartSLOduration=1.383457983 podStartE2EDuration="3.165308299s" totalImagesPullingTime="1.781850316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:36:10.164724299 +0000 UTC m=+482.709673500" watchObservedRunningTime="2026-07-23 01:36:10.165308299 +0000 UTC m=+482.710257471" Jul 23 01:36:11.154395 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:11.154366 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" Jul 23 01:36:22.161757 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:22.161723 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" Jul 23 01:36:31.612558 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.612507 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw"] Jul 23 01:36:31.613002 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.612774 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" podUID="2e1b5b8c-f233-42d3-881d-b14da3db631e" containerName="manager" containerID="cri-o://9393515a03e27872b6f7c5ace96c8f480e84726242ecf5dc834e3600e80807f9" gracePeriod=2 Jul 23 01:36:31.619685 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.619629 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw"] Jul 23 01:36:31.622972 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:36:31.622931 2572 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2e1b5b8c-f233-42d3-881d-b14da3db631e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-npbfw\": pods \"limitador-operator-controller-manager-859d7b78bd-npbfw\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" Jul 23 01:36:31.638920 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.638888 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7"] Jul 23 01:36:31.639314 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.639281 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1b5b8c-f233-42d3-881d-b14da3db631e" containerName="manager" Jul 23 01:36:31.639314 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.639305 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1b5b8c-f233-42d3-881d-b14da3db631e" containerName="manager" Jul 23 01:36:31.639475 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.639403 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1b5b8c-f233-42d3-881d-b14da3db631e" containerName="manager" Jul 23 01:36:31.642664 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.642644 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7" Jul 23 01:36:31.644850 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:36:31.644821 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-npbfw\" is forbidden: User \"system:node:ip-10-0-131-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-117.ec2.internal' and this object" podUID="2e1b5b8c-f233-42d3-881d-b14da3db631e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" Jul 23 01:36:31.652992 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.652966 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7"] Jul 23 01:36:31.696996 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.696958 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrlt\" (UniqueName: \"kubernetes.io/projected/860a10d0-dde0-48a4-b4fa-9923f6b15e46-kube-api-access-8zrlt\") pod \"limitador-operator-controller-manager-859d7b78bd-zrff7\" (UID: \"860a10d0-dde0-48a4-b4fa-9923f6b15e46\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7" Jul 23 01:36:31.798203 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.798170 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrlt\" (UniqueName: \"kubernetes.io/projected/860a10d0-dde0-48a4-b4fa-9923f6b15e46-kube-api-access-8zrlt\") pod \"limitador-operator-controller-manager-859d7b78bd-zrff7\" (UID: \"860a10d0-dde0-48a4-b4fa-9923f6b15e46\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7" Jul 23 01:36:31.806967 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.806934 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrlt\" (UniqueName: \"kubernetes.io/projected/860a10d0-dde0-48a4-b4fa-9923f6b15e46-kube-api-access-8zrlt\") pod \"limitador-operator-controller-manager-859d7b78bd-zrff7\" (UID: \"860a10d0-dde0-48a4-b4fa-9923f6b15e46\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7" Jul 23 01:36:31.858818 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.858781 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" Jul 23 01:36:31.860914 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:36:31.860889 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-npbfw\" is forbidden: User \"system:node:ip-10-0-131-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-117.ec2.internal' and this object" podUID="2e1b5b8c-f233-42d3-881d-b14da3db631e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" Jul 23 01:36:31.898560 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.898479 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr9kh\" (UniqueName: \"kubernetes.io/projected/2e1b5b8c-f233-42d3-881d-b14da3db631e-kube-api-access-qr9kh\") pod \"2e1b5b8c-f233-42d3-881d-b14da3db631e\" (UID: \"2e1b5b8c-f233-42d3-881d-b14da3db631e\") " Jul 23 01:36:31.900507 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.900480 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1b5b8c-f233-42d3-881d-b14da3db631e-kube-api-access-qr9kh" (OuterVolumeSpecName: "kube-api-access-qr9kh") pod "2e1b5b8c-f233-42d3-881d-b14da3db631e" (UID: "2e1b5b8c-f233-42d3-881d-b14da3db631e"). InnerVolumeSpecName "kube-api-access-qr9kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:36:31.975371 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.975338 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1b5b8c-f233-42d3-881d-b14da3db631e" path="/var/lib/kubelet/pods/2e1b5b8c-f233-42d3-881d-b14da3db631e/volumes" Jul 23 01:36:31.993342 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.993316 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7" Jul 23 01:36:31.999521 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:31.999494 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr9kh\" (UniqueName: \"kubernetes.io/projected/2e1b5b8c-f233-42d3-881d-b14da3db631e-kube-api-access-qr9kh\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:36:32.136306 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:32.136274 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7"] Jul 23 01:36:32.138162 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:36:32.138132 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860a10d0_dde0_48a4_b4fa_9923f6b15e46.slice/crio-6f4981901fadd0eee347b59d399f20105e2a692f1492650a2c03a149033a8cc1 WatchSource:0}: Error finding container 6f4981901fadd0eee347b59d399f20105e2a692f1492650a2c03a149033a8cc1: Status 404 returned error can't find the container with id 6f4981901fadd0eee347b59d399f20105e2a692f1492650a2c03a149033a8cc1 Jul 23 01:36:32.257753 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:32.257716 2572 generic.go:356] "Generic (PLEG): container finished" podID="2e1b5b8c-f233-42d3-881d-b14da3db631e" containerID="9393515a03e27872b6f7c5ace96c8f480e84726242ecf5dc834e3600e80807f9" exitCode=0 Jul 23 01:36:32.257931 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:32.257757 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-npbfw" Jul 23 01:36:32.257931 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:32.257820 2572 scope.go:117] "RemoveContainer" containerID="9393515a03e27872b6f7c5ace96c8f480e84726242ecf5dc834e3600e80807f9" Jul 23 01:36:32.259601 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:32.259468 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7" event={"ID":"860a10d0-dde0-48a4-b4fa-9923f6b15e46","Type":"ContainerStarted","Data":"ebe4189570aafb52b93bb413914def3c4dac33231aadadd84397ea98b84e3145"} Jul 23 01:36:32.259601 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:32.259523 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7" event={"ID":"860a10d0-dde0-48a4-b4fa-9923f6b15e46","Type":"ContainerStarted","Data":"6f4981901fadd0eee347b59d399f20105e2a692f1492650a2c03a149033a8cc1"} Jul 23 01:36:32.259601 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:32.259568 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7" Jul 23 01:36:32.266362 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:32.266345 2572 scope.go:117] "RemoveContainer" containerID="9393515a03e27872b6f7c5ace96c8f480e84726242ecf5dc834e3600e80807f9" Jul 23 01:36:32.266660 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:36:32.266615 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9393515a03e27872b6f7c5ace96c8f480e84726242ecf5dc834e3600e80807f9\": container with ID starting with 9393515a03e27872b6f7c5ace96c8f480e84726242ecf5dc834e3600e80807f9 not found: ID does not exist" containerID="9393515a03e27872b6f7c5ace96c8f480e84726242ecf5dc834e3600e80807f9" Jul 23 01:36:32.266748 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:32.266655 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9393515a03e27872b6f7c5ace96c8f480e84726242ecf5dc834e3600e80807f9"} err="failed to get container status \"9393515a03e27872b6f7c5ace96c8f480e84726242ecf5dc834e3600e80807f9\": rpc error: code = NotFound desc = could not find container \"9393515a03e27872b6f7c5ace96c8f480e84726242ecf5dc834e3600e80807f9\": container with ID starting with 9393515a03e27872b6f7c5ace96c8f480e84726242ecf5dc834e3600e80807f9 not found: ID does not exist" Jul 23 01:36:32.285177 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:32.285118 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7" podStartSLOduration=1.28510397 podStartE2EDuration="1.28510397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:36:32.283897015 +0000 UTC m=+504.828846214" watchObservedRunningTime="2026-07-23 01:36:32.28510397 +0000 UTC m=+504.830053142" Jul 23 01:36:34.221628 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.221580 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f85c8bbb6-xckr4" podUID="da94f987-03b0-45ec-822a-88c01b5071e4" containerName="console" containerID="cri-o://bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608" gracePeriod=15 Jul 23 01:36:34.476813 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.476740 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:36:34.517071 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.517039 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-trusted-ca-bundle\") pod \"da94f987-03b0-45ec-822a-88c01b5071e4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " Jul 23 01:36:34.517202 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.517078 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da94f987-03b0-45ec-822a-88c01b5071e4-console-oauth-config\") pod \"da94f987-03b0-45ec-822a-88c01b5071e4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " Jul 23 01:36:34.517202 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.517129 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5js7c\" (UniqueName: \"kubernetes.io/projected/da94f987-03b0-45ec-822a-88c01b5071e4-kube-api-access-5js7c\") pod \"da94f987-03b0-45ec-822a-88c01b5071e4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " Jul 23 01:36:34.517202 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.517192 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-service-ca\") pod \"da94f987-03b0-45ec-822a-88c01b5071e4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " Jul 23 01:36:34.517353 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.517221 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-console-config\") pod \"da94f987-03b0-45ec-822a-88c01b5071e4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " Jul 23 01:36:34.517353 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.517253 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da94f987-03b0-45ec-822a-88c01b5071e4-console-serving-cert\") pod \"da94f987-03b0-45ec-822a-88c01b5071e4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " Jul 23 01:36:34.517353 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.517279 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-oauth-serving-cert\") pod \"da94f987-03b0-45ec-822a-88c01b5071e4\" (UID: \"da94f987-03b0-45ec-822a-88c01b5071e4\") " Jul 23 01:36:34.517542 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.517466 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "da94f987-03b0-45ec-822a-88c01b5071e4" (UID: "da94f987-03b0-45ec-822a-88c01b5071e4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:36:34.517769 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.517725 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-service-ca" (OuterVolumeSpecName: "service-ca") pod "da94f987-03b0-45ec-822a-88c01b5071e4" (UID: "da94f987-03b0-45ec-822a-88c01b5071e4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:36:34.517876 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.517769 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "da94f987-03b0-45ec-822a-88c01b5071e4" (UID: "da94f987-03b0-45ec-822a-88c01b5071e4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:36:34.517876 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.517766 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-console-config" (OuterVolumeSpecName: "console-config") pod "da94f987-03b0-45ec-822a-88c01b5071e4" (UID: "da94f987-03b0-45ec-822a-88c01b5071e4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:36:34.519448 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.519417 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da94f987-03b0-45ec-822a-88c01b5071e4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "da94f987-03b0-45ec-822a-88c01b5071e4" (UID: "da94f987-03b0-45ec-822a-88c01b5071e4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:36:34.519448 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.519418 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da94f987-03b0-45ec-822a-88c01b5071e4-kube-api-access-5js7c" (OuterVolumeSpecName: "kube-api-access-5js7c") pod "da94f987-03b0-45ec-822a-88c01b5071e4" (UID: "da94f987-03b0-45ec-822a-88c01b5071e4"). InnerVolumeSpecName "kube-api-access-5js7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:36:34.519739 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.519720 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da94f987-03b0-45ec-822a-88c01b5071e4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "da94f987-03b0-45ec-822a-88c01b5071e4" (UID: "da94f987-03b0-45ec-822a-88c01b5071e4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:36:34.618071 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.618031 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-console-config\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:36:34.618071 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.618063 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da94f987-03b0-45ec-822a-88c01b5071e4-console-serving-cert\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:36:34.618071 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.618075 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-oauth-serving-cert\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:36:34.618311 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.618088 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-trusted-ca-bundle\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:36:34.618311 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.618101 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da94f987-03b0-45ec-822a-88c01b5071e4-console-oauth-config\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:36:34.618311 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.618116 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5js7c\" (UniqueName: \"kubernetes.io/projected/da94f987-03b0-45ec-822a-88c01b5071e4-kube-api-access-5js7c\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:36:34.618311 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:34.618128 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da94f987-03b0-45ec-822a-88c01b5071e4-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:36:35.276783 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:35.276745 2572 generic.go:356] "Generic (PLEG): container finished" podID="da94f987-03b0-45ec-822a-88c01b5071e4" containerID="bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608" exitCode=0 Jul 23 01:36:35.277182 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:35.276788 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f85c8bbb6-xckr4" Jul 23 01:36:35.277182 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:35.276840 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f85c8bbb6-xckr4" event={"ID":"da94f987-03b0-45ec-822a-88c01b5071e4","Type":"ContainerDied","Data":"bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608"} Jul 23 01:36:35.277182 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:35.276888 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f85c8bbb6-xckr4" event={"ID":"da94f987-03b0-45ec-822a-88c01b5071e4","Type":"ContainerDied","Data":"3d6231aa41e3029b071a7abd02c0eb23c76e8aed87a89d5f3a9b5dba37ba6b86"} Jul 23 01:36:35.277182 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:35.276910 2572 scope.go:117] "RemoveContainer" containerID="bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608" Jul 23 01:36:35.285740 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:35.285725 2572 scope.go:117] "RemoveContainer" containerID="bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608" Jul 23 01:36:35.286024 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:36:35.286002 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608\": container with ID starting with bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608 not found: ID does not exist" containerID="bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608" Jul 23 01:36:35.286128 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:35.286029 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608"} err="failed to get container status \"bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608\": rpc error: code = NotFound desc = could not find container \"bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608\": container with ID starting with bf377ee99a194dc853c4b4515d63eaa12ea23ea82661b3d9d6ce2de3b98fc608 not found: ID does not exist" Jul 23 01:36:35.297968 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:35.297940 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f85c8bbb6-xckr4"] Jul 23 01:36:35.301621 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:35.301598 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f85c8bbb6-xckr4"] Jul 23 01:36:35.975699 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:35.975661 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da94f987-03b0-45ec-822a-88c01b5071e4" path="/var/lib/kubelet/pods/da94f987-03b0-45ec-822a-88c01b5071e4/volumes" Jul 23 01:36:43.268521 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:36:43.268486 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrff7" Jul 23 01:37:16.431770 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.431733 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hcjqj"] Jul 23 01:37:16.432216 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.432055 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da94f987-03b0-45ec-822a-88c01b5071e4" containerName="console" Jul 23 01:37:16.432216 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.432065 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="da94f987-03b0-45ec-822a-88c01b5071e4" containerName="console" Jul 23 01:37:16.432216 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.432124 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="da94f987-03b0-45ec-822a-88c01b5071e4" containerName="console" Jul 23 01:37:16.436505 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.436484 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" Jul 23 01:37:16.438868 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.438845 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 23 01:37:16.439010 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.438842 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-x2gdz\"" Jul 23 01:37:16.442088 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.442061 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hcjqj"] Jul 23 01:37:16.532905 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.532870 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hcjqj"] Jul 23 01:37:16.557854 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.557819 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd2tw\" (UniqueName: \"kubernetes.io/projected/4edd23b1-c59c-4c19-9c6d-987e15f31ecf-kube-api-access-pd2tw\") pod \"limitador-limitador-64c94fd64f-hcjqj\" (UID: \"4edd23b1-c59c-4c19-9c6d-987e15f31ecf\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" Jul 23 01:37:16.557999 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.557867 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4edd23b1-c59c-4c19-9c6d-987e15f31ecf-config-file\") pod \"limitador-limitador-64c94fd64f-hcjqj\" (UID: \"4edd23b1-c59c-4c19-9c6d-987e15f31ecf\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" Jul 23 01:37:16.658759 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.658727 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2tw\" (UniqueName: \"kubernetes.io/projected/4edd23b1-c59c-4c19-9c6d-987e15f31ecf-kube-api-access-pd2tw\") pod \"limitador-limitador-64c94fd64f-hcjqj\" (UID: \"4edd23b1-c59c-4c19-9c6d-987e15f31ecf\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" Jul 23 01:37:16.658938 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.658773 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4edd23b1-c59c-4c19-9c6d-987e15f31ecf-config-file\") pod \"limitador-limitador-64c94fd64f-hcjqj\" (UID: \"4edd23b1-c59c-4c19-9c6d-987e15f31ecf\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" Jul 23 01:37:16.659346 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.659327 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4edd23b1-c59c-4c19-9c6d-987e15f31ecf-config-file\") pod \"limitador-limitador-64c94fd64f-hcjqj\" (UID: \"4edd23b1-c59c-4c19-9c6d-987e15f31ecf\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" Jul 23 01:37:16.666176 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.666151 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2tw\" (UniqueName: \"kubernetes.io/projected/4edd23b1-c59c-4c19-9c6d-987e15f31ecf-kube-api-access-pd2tw\") pod \"limitador-limitador-64c94fd64f-hcjqj\" (UID: \"4edd23b1-c59c-4c19-9c6d-987e15f31ecf\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" Jul 23 01:37:16.747695 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.747600 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" Jul 23 01:37:16.888850 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:16.888820 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hcjqj"] Jul 23 01:37:16.890309 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:37:16.890282 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4edd23b1_c59c_4c19_9c6d_987e15f31ecf.slice/crio-83042dbddce53e28489d609d6d0352e0a429053952d828afaae799152c1f7eb6 WatchSource:0}: Error finding container 83042dbddce53e28489d609d6d0352e0a429053952d828afaae799152c1f7eb6: Status 404 returned error can't find the container with id 83042dbddce53e28489d609d6d0352e0a429053952d828afaae799152c1f7eb6 Jul 23 01:37:17.482092 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:17.482055 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" event={"ID":"4edd23b1-c59c-4c19-9c6d-987e15f31ecf","Type":"ContainerStarted","Data":"83042dbddce53e28489d609d6d0352e0a429053952d828afaae799152c1f7eb6"} Jul 23 01:37:20.503422 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:20.503381 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" event={"ID":"4edd23b1-c59c-4c19-9c6d-987e15f31ecf","Type":"ContainerStarted","Data":"cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8"} Jul 23 01:37:20.503880 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:20.503533 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" Jul 23 01:37:20.519562 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:20.519511 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" podStartSLOduration=1.870352985 podStartE2EDuration="4.519496738s" totalImagesPullingTime="2.649143753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:37:20.517443009 +0000 UTC m=+553.062392321" watchObservedRunningTime="2026-07-23 01:37:20.519496738 +0000 UTC m=+553.064445911" Jul 23 01:37:31.305730 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:31.305689 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hcjqj"] Jul 23 01:37:31.306211 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:31.305946 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" podUID="4edd23b1-c59c-4c19-9c6d-987e15f31ecf" containerName="limitador" containerID="cri-o://cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8" gracePeriod=30 Jul 23 01:37:31.306863 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:31.306693 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" Jul 23 01:37:31.860621 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:31.860593 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" Jul 23 01:37:31.988224 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:31.988138 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd2tw\" (UniqueName: \"kubernetes.io/projected/4edd23b1-c59c-4c19-9c6d-987e15f31ecf-kube-api-access-pd2tw\") pod \"4edd23b1-c59c-4c19-9c6d-987e15f31ecf\" (UID: \"4edd23b1-c59c-4c19-9c6d-987e15f31ecf\") " Jul 23 01:37:31.988224 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:31.988208 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4edd23b1-c59c-4c19-9c6d-987e15f31ecf-config-file\") pod \"4edd23b1-c59c-4c19-9c6d-987e15f31ecf\" (UID: \"4edd23b1-c59c-4c19-9c6d-987e15f31ecf\") " Jul 23 01:37:31.988614 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:31.988584 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4edd23b1-c59c-4c19-9c6d-987e15f31ecf-config-file" (OuterVolumeSpecName: "config-file") pod "4edd23b1-c59c-4c19-9c6d-987e15f31ecf" (UID: "4edd23b1-c59c-4c19-9c6d-987e15f31ecf"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:37:31.990491 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:31.990457 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edd23b1-c59c-4c19-9c6d-987e15f31ecf-kube-api-access-pd2tw" (OuterVolumeSpecName: "kube-api-access-pd2tw") pod "4edd23b1-c59c-4c19-9c6d-987e15f31ecf" (UID: "4edd23b1-c59c-4c19-9c6d-987e15f31ecf"). InnerVolumeSpecName "kube-api-access-pd2tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:37:32.088998 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:32.088963 2572 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4edd23b1-c59c-4c19-9c6d-987e15f31ecf-config-file\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:37:32.088998 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:32.088994 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd2tw\" (UniqueName: \"kubernetes.io/projected/4edd23b1-c59c-4c19-9c6d-987e15f31ecf-kube-api-access-pd2tw\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:37:32.562237 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:32.562201 2572 generic.go:356] "Generic (PLEG): container finished" podID="4edd23b1-c59c-4c19-9c6d-987e15f31ecf" containerID="cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8" exitCode=0 Jul 23 01:37:32.562664 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:32.562243 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" Jul 23 01:37:32.562664 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:32.562278 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" event={"ID":"4edd23b1-c59c-4c19-9c6d-987e15f31ecf","Type":"ContainerDied","Data":"cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8"} Jul 23 01:37:32.562664 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:32.562332 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hcjqj" event={"ID":"4edd23b1-c59c-4c19-9c6d-987e15f31ecf","Type":"ContainerDied","Data":"83042dbddce53e28489d609d6d0352e0a429053952d828afaae799152c1f7eb6"} Jul 23 01:37:32.562664 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:32.562358 2572 scope.go:117] "RemoveContainer" containerID="cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8" Jul 23 01:37:32.571070 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:32.571053 2572 scope.go:117] "RemoveContainer" containerID="cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8" Jul 23 01:37:32.571332 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:37:32.571314 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8\": container with ID starting with cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8 not found: ID does not exist" containerID="cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8" Jul 23 01:37:32.571374 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:32.571342 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8"} err="failed to get container status \"cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8\": rpc error: code = NotFound desc = could not find container \"cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8\": container with ID starting with cf1c9cb5060273cd01a9514e2f8d10f63ddd72562272dc4720a44cd988a1e3d8 not found: ID does not exist" Jul 23 01:37:32.582597 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:32.582574 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hcjqj"] Jul 23 01:37:32.586480 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:32.586459 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hcjqj"] Jul 23 01:37:33.881768 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:33.881735 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v"] Jul 23 01:37:33.882133 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:33.882074 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4edd23b1-c59c-4c19-9c6d-987e15f31ecf" containerName="limitador" Jul 23 01:37:33.882133 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:33.882087 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4edd23b1-c59c-4c19-9c6d-987e15f31ecf" containerName="limitador" Jul 23 01:37:33.882208 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:33.882147 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4edd23b1-c59c-4c19-9c6d-987e15f31ecf" containerName="limitador" Jul 23 01:37:33.886246 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:33.886228 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" Jul 23 01:37:33.888656 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:33.888611 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 01:37:33.888776 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:33.888678 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 23 01:37:33.888776 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:33.888687 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 01:37:33.889619 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:33.889594 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-xr9xx\"" Jul 23 01:37:33.892066 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:33.892046 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v"] Jul 23 01:37:33.976452 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:33.976417 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4edd23b1-c59c-4c19-9c6d-987e15f31ecf" path="/var/lib/kubelet/pods/4edd23b1-c59c-4c19-9c6d-987e15f31ecf/volumes" Jul 23 01:37:34.005449 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:34.005418 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw67r\" (UniqueName: \"kubernetes.io/projected/20ead147-5678-4a18-bc75-c4ecf689b5b4-kube-api-access-qw67r\") pod \"postgres-69fb4594cf-s6q4v\" (UID: \"20ead147-5678-4a18-bc75-c4ecf689b5b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" Jul 23 01:37:34.005604 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:34.005539 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/20ead147-5678-4a18-bc75-c4ecf689b5b4-data\") pod \"postgres-69fb4594cf-s6q4v\" (UID: \"20ead147-5678-4a18-bc75-c4ecf689b5b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" Jul 23 01:37:34.106194 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:34.106157 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/20ead147-5678-4a18-bc75-c4ecf689b5b4-data\") pod \"postgres-69fb4594cf-s6q4v\" (UID: \"20ead147-5678-4a18-bc75-c4ecf689b5b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" Jul 23 01:37:34.106361 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:34.106206 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw67r\" (UniqueName: \"kubernetes.io/projected/20ead147-5678-4a18-bc75-c4ecf689b5b4-kube-api-access-qw67r\") pod \"postgres-69fb4594cf-s6q4v\" (UID: \"20ead147-5678-4a18-bc75-c4ecf689b5b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" Jul 23 01:37:34.106528 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:34.106497 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/20ead147-5678-4a18-bc75-c4ecf689b5b4-data\") pod \"postgres-69fb4594cf-s6q4v\" (UID: \"20ead147-5678-4a18-bc75-c4ecf689b5b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" Jul 23 01:37:34.113470 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:34.113438 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw67r\" (UniqueName: \"kubernetes.io/projected/20ead147-5678-4a18-bc75-c4ecf689b5b4-kube-api-access-qw67r\") pod \"postgres-69fb4594cf-s6q4v\" (UID: \"20ead147-5678-4a18-bc75-c4ecf689b5b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" Jul 23 01:37:34.197172 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:34.197078 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" Jul 23 01:37:34.329229 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:34.329205 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v"] Jul 23 01:37:34.331606 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:37:34.331580 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ead147_5678_4a18_bc75_c4ecf689b5b4.slice/crio-bd1e990213df9ffb2bedcd641c7f71daacc4f5cf9a9f6c393a6a55d1d645ea80 WatchSource:0}: Error finding container bd1e990213df9ffb2bedcd641c7f71daacc4f5cf9a9f6c393a6a55d1d645ea80: Status 404 returned error can't find the container with id bd1e990213df9ffb2bedcd641c7f71daacc4f5cf9a9f6c393a6a55d1d645ea80 Jul 23 01:37:34.574590 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:34.574505 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" event={"ID":"20ead147-5678-4a18-bc75-c4ecf689b5b4","Type":"ContainerStarted","Data":"bd1e990213df9ffb2bedcd641c7f71daacc4f5cf9a9f6c393a6a55d1d645ea80"} Jul 23 01:37:40.614528 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:40.614487 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" event={"ID":"20ead147-5678-4a18-bc75-c4ecf689b5b4","Type":"ContainerStarted","Data":"02b167c5240c21098c38b1a1beb29dbc6998b2f84bd0d7371e759984b27cc0b5"} Jul 23 01:37:40.614969 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:40.614609 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" Jul 23 01:37:40.629369 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:40.629308 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" podStartSLOduration=2.273858035 podStartE2EDuration="7.629288046s" totalImagesPullingTime="5.355430011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:37:40.627670155 +0000 UTC m=+573.172619347" watchObservedRunningTime="2026-07-23 01:37:40.629288046 +0000 UTC m=+573.174237222" Jul 23 01:37:46.649397 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:46.649368 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6q4v" Jul 23 01:37:54.077027 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.076991 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cc8bb7484-d75b7"] Jul 23 01:37:54.097892 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.097855 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc8bb7484-d75b7"] Jul 23 01:37:54.098064 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.097973 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc8bb7484-d75b7" Jul 23 01:37:54.100268 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.100241 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 01:37:54.100268 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.100241 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 01:37:54.100471 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.100254 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4hpt5\"" Jul 23 01:37:54.181130 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.181086 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkvsn\" (UniqueName: \"kubernetes.io/projected/12416829-7189-4f06-a788-7f485b03a418-kube-api-access-wkvsn\") pod \"maas-controller-cc8bb7484-d75b7\" (UID: \"12416829-7189-4f06-a788-7f485b03a418\") " pod="opendatahub/maas-controller-cc8bb7484-d75b7" Jul 23 01:37:54.181302 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.181140 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12416829-7189-4f06-a788-7f485b03a418-webhook-cert\") pod \"maas-controller-cc8bb7484-d75b7\" (UID: \"12416829-7189-4f06-a788-7f485b03a418\") " pod="opendatahub/maas-controller-cc8bb7484-d75b7" Jul 23 01:37:54.281783 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.281739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvsn\" (UniqueName: \"kubernetes.io/projected/12416829-7189-4f06-a788-7f485b03a418-kube-api-access-wkvsn\") pod \"maas-controller-cc8bb7484-d75b7\" (UID: \"12416829-7189-4f06-a788-7f485b03a418\") " pod="opendatahub/maas-controller-cc8bb7484-d75b7" Jul 23 01:37:54.281899 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.281796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12416829-7189-4f06-a788-7f485b03a418-webhook-cert\") pod \"maas-controller-cc8bb7484-d75b7\" (UID: \"12416829-7189-4f06-a788-7f485b03a418\") " pod="opendatahub/maas-controller-cc8bb7484-d75b7" Jul 23 01:37:54.284454 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.284429 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12416829-7189-4f06-a788-7f485b03a418-webhook-cert\") pod \"maas-controller-cc8bb7484-d75b7\" (UID: \"12416829-7189-4f06-a788-7f485b03a418\") " pod="opendatahub/maas-controller-cc8bb7484-d75b7" Jul 23 01:37:54.289812 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.289790 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvsn\" (UniqueName: \"kubernetes.io/projected/12416829-7189-4f06-a788-7f485b03a418-kube-api-access-wkvsn\") pod \"maas-controller-cc8bb7484-d75b7\" (UID: \"12416829-7189-4f06-a788-7f485b03a418\") " pod="opendatahub/maas-controller-cc8bb7484-d75b7" Jul 23 01:37:54.409374 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.409340 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc8bb7484-d75b7" Jul 23 01:37:54.537148 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.537122 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc8bb7484-d75b7"] Jul 23 01:37:54.538978 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:37:54.538948 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12416829_7189_4f06_a788_7f485b03a418.slice/crio-1b4dd9393d228b8a8235761fdf4d5f88e987b1c28057f3422c84242324f0dc03 WatchSource:0}: Error finding container 1b4dd9393d228b8a8235761fdf4d5f88e987b1c28057f3422c84242324f0dc03: Status 404 returned error can't find the container with id 1b4dd9393d228b8a8235761fdf4d5f88e987b1c28057f3422c84242324f0dc03 Jul 23 01:37:54.683735 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:54.683627 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc8bb7484-d75b7" event={"ID":"12416829-7189-4f06-a788-7f485b03a418","Type":"ContainerStarted","Data":"1b4dd9393d228b8a8235761fdf4d5f88e987b1c28057f3422c84242324f0dc03"} Jul 23 01:37:57.701874 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:57.701835 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc8bb7484-d75b7" event={"ID":"12416829-7189-4f06-a788-7f485b03a418","Type":"ContainerStarted","Data":"530b05d5c9986a78ac3ff28f9ff54d55e26d1802809bd2bca1bae4f2adf456e4"} Jul 23 01:37:57.702262 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:57.701900 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cc8bb7484-d75b7" Jul 23 01:37:57.727759 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:37:57.727705 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cc8bb7484-d75b7" podStartSLOduration=1.259113846 podStartE2EDuration="3.727688127s" totalImagesPullingTime="2.468574281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:37:57.726310432 +0000 UTC m=+590.271259634" watchObservedRunningTime="2026-07-23 01:37:57.727688127 +0000 UTC m=+590.272637299" Jul 23 01:38:08.708781 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:08.708750 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cc8bb7484-d75b7" Jul 23 01:38:30.234354 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.234272 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-4whc6"] Jul 23 01:38:30.237867 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.237850 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" Jul 23 01:38:30.240208 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.240182 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 23 01:38:30.240372 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.240353 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-42wv9\"" Jul 23 01:38:30.247315 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.247281 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-4whc6"] Jul 23 01:38:30.386931 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.386884 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe40b256-bab4-48de-812f-f22624468832-plugins-config-volume\") pod \"payload-processing-5bf47cd988-4whc6\" (UID: \"fe40b256-bab4-48de-812f-f22624468832\") " pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" Jul 23 01:38:30.386931 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.386925 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvgbt\" (UniqueName: \"kubernetes.io/projected/fe40b256-bab4-48de-812f-f22624468832-kube-api-access-gvgbt\") pod \"payload-processing-5bf47cd988-4whc6\" (UID: \"fe40b256-bab4-48de-812f-f22624468832\") " pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" Jul 23 01:38:30.488062 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.487978 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe40b256-bab4-48de-812f-f22624468832-plugins-config-volume\") pod \"payload-processing-5bf47cd988-4whc6\" (UID: \"fe40b256-bab4-48de-812f-f22624468832\") " pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" Jul 23 01:38:30.488062 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.488017 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvgbt\" (UniqueName: \"kubernetes.io/projected/fe40b256-bab4-48de-812f-f22624468832-kube-api-access-gvgbt\") pod \"payload-processing-5bf47cd988-4whc6\" (UID: \"fe40b256-bab4-48de-812f-f22624468832\") " pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" Jul 23 01:38:30.488745 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.488725 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe40b256-bab4-48de-812f-f22624468832-plugins-config-volume\") pod \"payload-processing-5bf47cd988-4whc6\" (UID: \"fe40b256-bab4-48de-812f-f22624468832\") " pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" Jul 23 01:38:30.495776 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.495747 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvgbt\" (UniqueName: \"kubernetes.io/projected/fe40b256-bab4-48de-812f-f22624468832-kube-api-access-gvgbt\") pod \"payload-processing-5bf47cd988-4whc6\" (UID: \"fe40b256-bab4-48de-812f-f22624468832\") " pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" Jul 23 01:38:30.550403 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.550364 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" Jul 23 01:38:30.694131 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.694095 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-4whc6"] Jul 23 01:38:30.698398 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:38:30.698370 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe40b256_bab4_48de_812f_f22624468832.slice/crio-f5db8a112a725a19cfdfbb9a1518559ee042b6d1f8bfbf86db663effb93bce03 WatchSource:0}: Error finding container f5db8a112a725a19cfdfbb9a1518559ee042b6d1f8bfbf86db663effb93bce03: Status 404 returned error can't find the container with id f5db8a112a725a19cfdfbb9a1518559ee042b6d1f8bfbf86db663effb93bce03 Jul 23 01:38:30.700509 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.700488 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 01:38:30.867591 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:30.867554 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" event={"ID":"fe40b256-bab4-48de-812f-f22624468832","Type":"ContainerStarted","Data":"f5db8a112a725a19cfdfbb9a1518559ee042b6d1f8bfbf86db663effb93bce03"} Jul 23 01:38:32.881017 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:32.880978 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" event={"ID":"fe40b256-bab4-48de-812f-f22624468832","Type":"ContainerStarted","Data":"f13865b1a467adb6b77f6eebddf8a45665184748f957de6e188d22dde40a1297"} Jul 23 01:38:32.881391 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:32.881048 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" Jul 23 01:38:32.896937 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:32.896892 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" podStartSLOduration=1.3484432659999999 podStartE2EDuration="2.896878949s" totalImagesPullingTime="1.548435683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:38:32.894710293 +0000 UTC m=+625.439659491" watchObservedRunningTime="2026-07-23 01:38:32.896878949 +0000 UTC m=+625.441828122" Jul 23 01:38:43.888455 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:43.888417 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-4whc6" Jul 23 01:38:48.972331 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:48.972292 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8"] Jul 23 01:38:48.976194 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:48.976168 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:48.978473 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:48.978446 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ktgld\"" Jul 23 01:38:48.979226 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:48.979203 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 23 01:38:48.979318 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:48.979203 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 01:38:48.979318 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:48.979208 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 01:38:48.986382 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:48.986314 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8"] Jul 23 01:38:49.037595 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.037553 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290af6d9-abb7-4b07-9935-c4e4338a6c3f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.037752 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.037603 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290af6d9-abb7-4b07-9935-c4e4338a6c3f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.037752 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.037671 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290af6d9-abb7-4b07-9935-c4e4338a6c3f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.037752 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.037714 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290af6d9-abb7-4b07-9935-c4e4338a6c3f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.037752 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.037751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290af6d9-abb7-4b07-9935-c4e4338a6c3f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.037929 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.037787 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxh2\" (UniqueName: \"kubernetes.io/projected/290af6d9-abb7-4b07-9935-c4e4338a6c3f-kube-api-access-wfxh2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.138743 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.138704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290af6d9-abb7-4b07-9935-c4e4338a6c3f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.138940 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.138753 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290af6d9-abb7-4b07-9935-c4e4338a6c3f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.138940 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.138808 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxh2\" (UniqueName: \"kubernetes.io/projected/290af6d9-abb7-4b07-9935-c4e4338a6c3f-kube-api-access-wfxh2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.138940 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.138865 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290af6d9-abb7-4b07-9935-c4e4338a6c3f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.138940 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.138910 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290af6d9-abb7-4b07-9935-c4e4338a6c3f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.139159 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.139049 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290af6d9-abb7-4b07-9935-c4e4338a6c3f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.139215 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.139173 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290af6d9-abb7-4b07-9935-c4e4338a6c3f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.139275 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.139245 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290af6d9-abb7-4b07-9935-c4e4338a6c3f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.139328 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.139272 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290af6d9-abb7-4b07-9935-c4e4338a6c3f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.141083 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.141064 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290af6d9-abb7-4b07-9935-c4e4338a6c3f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.141328 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.141310 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290af6d9-abb7-4b07-9935-c4e4338a6c3f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.146139 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.146116 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxh2\" (UniqueName: \"kubernetes.io/projected/290af6d9-abb7-4b07-9935-c4e4338a6c3f-kube-api-access-wfxh2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8\" (UID: \"290af6d9-abb7-4b07-9935-c4e4338a6c3f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.288713 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.288577 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:49.429911 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.429877 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8"] Jul 23 01:38:49.431522 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:38:49.431475 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290af6d9_abb7_4b07_9935_c4e4338a6c3f.slice/crio-d3d6b92af830b75dd7cf3b250afce8efe0f94c657404af0743903745f04af2db WatchSource:0}: Error finding container d3d6b92af830b75dd7cf3b250afce8efe0f94c657404af0743903745f04af2db: Status 404 returned error can't find the container with id d3d6b92af830b75dd7cf3b250afce8efe0f94c657404af0743903745f04af2db Jul 23 01:38:49.978335 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:49.978270 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" event={"ID":"290af6d9-abb7-4b07-9935-c4e4338a6c3f","Type":"ContainerStarted","Data":"d3d6b92af830b75dd7cf3b250afce8efe0f94c657404af0743903745f04af2db"} Jul 23 01:38:50.978299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:50.978252 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" event={"ID":"290af6d9-abb7-4b07-9935-c4e4338a6c3f","Type":"ContainerStarted","Data":"3ccc48c939dc0580129002d01a639860e41411c2b25f2daac4108e44985aff24"} Jul 23 01:38:50.995015 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:50.994963 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" podStartSLOduration=1.86719738 podStartE2EDuration="2.994949017s" totalImagesPullingTime="1.127751637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:38:50.993681064 +0000 UTC m=+643.538630242" watchObservedRunningTime="2026-07-23 01:38:50.994949017 +0000 UTC m=+643.539898189" Jul 23 01:38:59.288861 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:59.288828 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:59.289269 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:59.288886 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:38:59.301500 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:38:59.301471 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:39:00.035232 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.035197 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8" Jul 23 01:39:00.588261 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.588222 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4"] Jul 23 01:39:00.595251 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.595211 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.599746 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.599717 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 23 01:39:00.603238 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.603207 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4"] Jul 23 01:39:00.644460 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.644427 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9027281c-43dc-4686-b027-dd88cce64e08-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.644621 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.644476 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9027281c-43dc-4686-b027-dd88cce64e08-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.644621 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.644505 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbhp\" (UniqueName: \"kubernetes.io/projected/9027281c-43dc-4686-b027-dd88cce64e08-kube-api-access-8bbhp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.644621 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.644566 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9027281c-43dc-4686-b027-dd88cce64e08-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.644621 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.644610 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9027281c-43dc-4686-b027-dd88cce64e08-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.644824 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.644703 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9027281c-43dc-4686-b027-dd88cce64e08-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.745406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.745367 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9027281c-43dc-4686-b027-dd88cce64e08-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.745406 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.745410 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9027281c-43dc-4686-b027-dd88cce64e08-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.745668 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.745455 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9027281c-43dc-4686-b027-dd88cce64e08-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.745668 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.745481 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9027281c-43dc-4686-b027-dd88cce64e08-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.745668 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.745498 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbhp\" (UniqueName: \"kubernetes.io/projected/9027281c-43dc-4686-b027-dd88cce64e08-kube-api-access-8bbhp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.745836 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.745677 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9027281c-43dc-4686-b027-dd88cce64e08-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.745836 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.745791 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9027281c-43dc-4686-b027-dd88cce64e08-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.745937 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.745873 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9027281c-43dc-4686-b027-dd88cce64e08-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.746009 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.745990 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9027281c-43dc-4686-b027-dd88cce64e08-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.747781 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.747756 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9027281c-43dc-4686-b027-dd88cce64e08-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.747987 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.747971 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9027281c-43dc-4686-b027-dd88cce64e08-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.758144 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.758119 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbhp\" (UniqueName: \"kubernetes.io/projected/9027281c-43dc-4686-b027-dd88cce64e08-kube-api-access-8bbhp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5csh4\" (UID: \"9027281c-43dc-4686-b027-dd88cce64e08\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:00.909517 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:00.909478 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:01.046062 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:01.046035 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4"] Jul 23 01:39:01.050191 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:39:01.047851 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9027281c_43dc_4686_b027_dd88cce64e08.slice/crio-758a4643b2c194291242f5c7afe4706e6ba4f1a93136a63184c388aede3503d0 WatchSource:0}: Error finding container 758a4643b2c194291242f5c7afe4706e6ba4f1a93136a63184c388aede3503d0: Status 404 returned error can't find the container with id 758a4643b2c194291242f5c7afe4706e6ba4f1a93136a63184c388aede3503d0 Jul 23 01:39:02.036226 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:02.036186 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" event={"ID":"9027281c-43dc-4686-b027-dd88cce64e08","Type":"ContainerStarted","Data":"a9ef20aa50cbaeb6704a56b0aa2526f9c812514c212cdbb9cabbc78eccaef82e"} Jul 23 01:39:02.036674 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:02.036256 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" event={"ID":"9027281c-43dc-4686-b027-dd88cce64e08","Type":"ContainerStarted","Data":"758a4643b2c194291242f5c7afe4706e6ba4f1a93136a63184c388aede3503d0"} Jul 23 01:39:02.052128 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:02.052078 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" podStartSLOduration=1.854904775 podStartE2EDuration="2.052063529s" totalImagesPullingTime="197.158754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:39:02.051426107 +0000 UTC m=+654.596375324" watchObservedRunningTime="2026-07-23 01:39:02.052063529 +0000 UTC m=+654.597012719" Jul 23 01:39:10.909761 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:10.909719 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:10.910157 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:10.909789 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:10.922457 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:10.922431 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:39:11.090833 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:39:11.090804 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5csh4" Jul 23 01:41:50.657238 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:50.657196 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cc8bb7484-d75b7"] Jul 23 01:41:50.657722 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:50.657491 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cc8bb7484-d75b7" podUID="12416829-7189-4f06-a788-7f485b03a418" containerName="manager" containerID="cri-o://530b05d5c9986a78ac3ff28f9ff54d55e26d1802809bd2bca1bae4f2adf456e4" gracePeriod=30 Jul 23 01:41:50.894935 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:50.894906 2572 generic.go:356] "Generic (PLEG): container finished" podID="12416829-7189-4f06-a788-7f485b03a418" containerID="530b05d5c9986a78ac3ff28f9ff54d55e26d1802809bd2bca1bae4f2adf456e4" exitCode=0 Jul 23 01:41:50.895055 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:50.894979 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc8bb7484-d75b7" event={"ID":"12416829-7189-4f06-a788-7f485b03a418","Type":"ContainerDied","Data":"530b05d5c9986a78ac3ff28f9ff54d55e26d1802809bd2bca1bae4f2adf456e4"} Jul 23 01:41:50.895055 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:50.895047 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc8bb7484-d75b7" event={"ID":"12416829-7189-4f06-a788-7f485b03a418","Type":"ContainerDied","Data":"1b4dd9393d228b8a8235761fdf4d5f88e987b1c28057f3422c84242324f0dc03"} Jul 23 01:41:50.895135 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:50.895064 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b4dd9393d228b8a8235761fdf4d5f88e987b1c28057f3422c84242324f0dc03" Jul 23 01:41:50.908366 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:50.908306 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc8bb7484-d75b7" Jul 23 01:41:51.011548 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:51.011512 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkvsn\" (UniqueName: \"kubernetes.io/projected/12416829-7189-4f06-a788-7f485b03a418-kube-api-access-wkvsn\") pod \"12416829-7189-4f06-a788-7f485b03a418\" (UID: \"12416829-7189-4f06-a788-7f485b03a418\") " Jul 23 01:41:51.011764 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:51.011615 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12416829-7189-4f06-a788-7f485b03a418-webhook-cert\") pod \"12416829-7189-4f06-a788-7f485b03a418\" (UID: \"12416829-7189-4f06-a788-7f485b03a418\") " Jul 23 01:41:51.013620 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:51.013594 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12416829-7189-4f06-a788-7f485b03a418-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "12416829-7189-4f06-a788-7f485b03a418" (UID: "12416829-7189-4f06-a788-7f485b03a418"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:41:51.013745 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:51.013708 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12416829-7189-4f06-a788-7f485b03a418-kube-api-access-wkvsn" (OuterVolumeSpecName: "kube-api-access-wkvsn") pod "12416829-7189-4f06-a788-7f485b03a418" (UID: "12416829-7189-4f06-a788-7f485b03a418"). InnerVolumeSpecName "kube-api-access-wkvsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:41:51.112647 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:51.112607 2572 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12416829-7189-4f06-a788-7f485b03a418-webhook-cert\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:41:51.112797 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:51.112658 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkvsn\" (UniqueName: \"kubernetes.io/projected/12416829-7189-4f06-a788-7f485b03a418-kube-api-access-wkvsn\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:41:51.899704 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:51.899669 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc8bb7484-d75b7" Jul 23 01:41:51.919870 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:51.919837 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cc8bb7484-d75b7"] Jul 23 01:41:51.923277 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:51.923252 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cc8bb7484-d75b7"] Jul 23 01:41:51.975580 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:41:51.975548 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12416829-7189-4f06-a788-7f485b03a418" path="/var/lib/kubelet/pods/12416829-7189-4f06-a788-7f485b03a418/volumes" Jul 23 01:42:52.698375 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.698336 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cc8bb7484-62ps5"] Jul 23 01:42:52.698879 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.698703 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12416829-7189-4f06-a788-7f485b03a418" containerName="manager" Jul 23 01:42:52.698879 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.698716 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="12416829-7189-4f06-a788-7f485b03a418" containerName="manager" Jul 23 01:42:52.698879 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.698794 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="12416829-7189-4f06-a788-7f485b03a418" containerName="manager" Jul 23 01:42:52.702042 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.702026 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc8bb7484-62ps5" Jul 23 01:42:52.705183 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.705120 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 01:42:52.705183 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.705161 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4hpt5\"" Jul 23 01:42:52.705364 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.705180 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 01:42:52.710579 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.710557 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc8bb7484-62ps5"] Jul 23 01:42:52.866499 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.866452 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r6tx\" (UniqueName: \"kubernetes.io/projected/2866217e-ac60-4435-b023-cc1b3cddc6a9-kube-api-access-5r6tx\") pod \"maas-controller-cc8bb7484-62ps5\" (UID: \"2866217e-ac60-4435-b023-cc1b3cddc6a9\") " pod="opendatahub/maas-controller-cc8bb7484-62ps5" Jul 23 01:42:52.866708 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.866572 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2866217e-ac60-4435-b023-cc1b3cddc6a9-webhook-cert\") pod \"maas-controller-cc8bb7484-62ps5\" (UID: \"2866217e-ac60-4435-b023-cc1b3cddc6a9\") " pod="opendatahub/maas-controller-cc8bb7484-62ps5" Jul 23 01:42:52.968087 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.967987 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r6tx\" (UniqueName: \"kubernetes.io/projected/2866217e-ac60-4435-b023-cc1b3cddc6a9-kube-api-access-5r6tx\") pod \"maas-controller-cc8bb7484-62ps5\" (UID: \"2866217e-ac60-4435-b023-cc1b3cddc6a9\") " pod="opendatahub/maas-controller-cc8bb7484-62ps5" Jul 23 01:42:52.968087 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.968084 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2866217e-ac60-4435-b023-cc1b3cddc6a9-webhook-cert\") pod \"maas-controller-cc8bb7484-62ps5\" (UID: \"2866217e-ac60-4435-b023-cc1b3cddc6a9\") " pod="opendatahub/maas-controller-cc8bb7484-62ps5" Jul 23 01:42:52.970556 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.970528 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2866217e-ac60-4435-b023-cc1b3cddc6a9-webhook-cert\") pod \"maas-controller-cc8bb7484-62ps5\" (UID: \"2866217e-ac60-4435-b023-cc1b3cddc6a9\") " pod="opendatahub/maas-controller-cc8bb7484-62ps5" Jul 23 01:42:52.975134 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:52.975115 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r6tx\" (UniqueName: \"kubernetes.io/projected/2866217e-ac60-4435-b023-cc1b3cddc6a9-kube-api-access-5r6tx\") pod \"maas-controller-cc8bb7484-62ps5\" (UID: \"2866217e-ac60-4435-b023-cc1b3cddc6a9\") " pod="opendatahub/maas-controller-cc8bb7484-62ps5" Jul 23 01:42:53.013081 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:53.013049 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc8bb7484-62ps5" Jul 23 01:42:53.144592 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:53.144562 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc8bb7484-62ps5"] Jul 23 01:42:53.146189 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:42:53.146160 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2866217e_ac60_4435_b023_cc1b3cddc6a9.slice/crio-7a668c11a20c63dd90328cc4a708f9c413c804a8329778ca79c19889baac8461 WatchSource:0}: Error finding container 7a668c11a20c63dd90328cc4a708f9c413c804a8329778ca79c19889baac8461: Status 404 returned error can't find the container with id 7a668c11a20c63dd90328cc4a708f9c413c804a8329778ca79c19889baac8461 Jul 23 01:42:53.212566 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:53.212530 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc8bb7484-62ps5" event={"ID":"2866217e-ac60-4435-b023-cc1b3cddc6a9","Type":"ContainerStarted","Data":"7a668c11a20c63dd90328cc4a708f9c413c804a8329778ca79c19889baac8461"} Jul 23 01:42:54.219699 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:54.219572 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc8bb7484-62ps5" event={"ID":"2866217e-ac60-4435-b023-cc1b3cddc6a9","Type":"ContainerStarted","Data":"f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301"} Jul 23 01:42:54.219699 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:54.219693 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cc8bb7484-62ps5" Jul 23 01:42:54.236774 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:42:54.236706 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cc8bb7484-62ps5" podStartSLOduration=1.4225190269999999 podStartE2EDuration="2.236683084s" totalImagesPullingTime="814.164057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:42:54.234919082 +0000 UTC m=+886.779868277" watchObservedRunningTime="2026-07-23 01:42:54.236683084 +0000 UTC m=+886.781632260" Jul 23 01:43:05.228063 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:05.227981 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cc8bb7484-62ps5" Jul 23 01:43:09.842412 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:09.842370 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cc8bb7484-62ps5"] Jul 23 01:43:09.842916 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:09.842602 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cc8bb7484-62ps5" podUID="2866217e-ac60-4435-b023-cc1b3cddc6a9" containerName="manager" containerID="cri-o://f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301" gracePeriod=30 Jul 23 01:43:10.091415 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.091390 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc8bb7484-62ps5" Jul 23 01:43:10.218031 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.217995 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r6tx\" (UniqueName: \"kubernetes.io/projected/2866217e-ac60-4435-b023-cc1b3cddc6a9-kube-api-access-5r6tx\") pod \"2866217e-ac60-4435-b023-cc1b3cddc6a9\" (UID: \"2866217e-ac60-4435-b023-cc1b3cddc6a9\") " Jul 23 01:43:10.218031 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.218037 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2866217e-ac60-4435-b023-cc1b3cddc6a9-webhook-cert\") pod \"2866217e-ac60-4435-b023-cc1b3cddc6a9\" (UID: \"2866217e-ac60-4435-b023-cc1b3cddc6a9\") " Jul 23 01:43:10.220076 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.220042 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2866217e-ac60-4435-b023-cc1b3cddc6a9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2866217e-ac60-4435-b023-cc1b3cddc6a9" (UID: "2866217e-ac60-4435-b023-cc1b3cddc6a9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:43:10.220195 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.220100 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2866217e-ac60-4435-b023-cc1b3cddc6a9-kube-api-access-5r6tx" (OuterVolumeSpecName: "kube-api-access-5r6tx") pod "2866217e-ac60-4435-b023-cc1b3cddc6a9" (UID: "2866217e-ac60-4435-b023-cc1b3cddc6a9"). InnerVolumeSpecName "kube-api-access-5r6tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:43:10.302062 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.302021 2572 generic.go:356] "Generic (PLEG): container finished" podID="2866217e-ac60-4435-b023-cc1b3cddc6a9" containerID="f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301" exitCode=0 Jul 23 01:43:10.302062 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.302065 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc8bb7484-62ps5" Jul 23 01:43:10.302317 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.302112 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc8bb7484-62ps5" event={"ID":"2866217e-ac60-4435-b023-cc1b3cddc6a9","Type":"ContainerDied","Data":"f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301"} Jul 23 01:43:10.302317 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.302174 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc8bb7484-62ps5" event={"ID":"2866217e-ac60-4435-b023-cc1b3cddc6a9","Type":"ContainerDied","Data":"7a668c11a20c63dd90328cc4a708f9c413c804a8329778ca79c19889baac8461"} Jul 23 01:43:10.302317 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.302203 2572 scope.go:117] "RemoveContainer" containerID="f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301" Jul 23 01:43:10.311254 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.311236 2572 scope.go:117] "RemoveContainer" containerID="f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301" Jul 23 01:43:10.311489 ip-10-0-131-117 kubenswrapper[2572]: E0723 01:43:10.311472 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301\": container with ID starting with f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301 not found: ID does not exist" containerID="f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301" Jul 23 01:43:10.311533 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.311498 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301"} err="failed to get container status \"f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301\": rpc error: code = NotFound desc = could not find container \"f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301\": container with ID starting with f334524b437b9ad3242de684b9b6de1c6764420ac98d06d8bfad8a44e43b7301 not found: ID does not exist" Jul 23 01:43:10.319144 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.319125 2572 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2866217e-ac60-4435-b023-cc1b3cddc6a9-webhook-cert\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:43:10.319229 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.319146 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r6tx\" (UniqueName: \"kubernetes.io/projected/2866217e-ac60-4435-b023-cc1b3cddc6a9-kube-api-access-5r6tx\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:43:10.322914 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.322890 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cc8bb7484-62ps5"] Jul 23 01:43:10.328339 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:10.328317 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cc8bb7484-62ps5"] Jul 23 01:43:11.096561 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.096526 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cc8bb7484-qss2q"] Jul 23 01:43:11.096969 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.096886 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2866217e-ac60-4435-b023-cc1b3cddc6a9" containerName="manager" Jul 23 01:43:11.096969 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.096899 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2866217e-ac60-4435-b023-cc1b3cddc6a9" containerName="manager" Jul 23 01:43:11.096969 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.096962 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2866217e-ac60-4435-b023-cc1b3cddc6a9" containerName="manager" Jul 23 01:43:11.101198 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.101177 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc8bb7484-qss2q" Jul 23 01:43:11.103707 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.103688 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 01:43:11.104665 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.104652 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 01:43:11.104721 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.104681 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4hpt5\"" Jul 23 01:43:11.110771 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.110749 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc8bb7484-qss2q"] Jul 23 01:43:11.226915 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.226878 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f761efa-e6f1-4dbe-8702-0c2a1bab3730-webhook-cert\") pod \"maas-controller-cc8bb7484-qss2q\" (UID: \"3f761efa-e6f1-4dbe-8702-0c2a1bab3730\") " pod="opendatahub/maas-controller-cc8bb7484-qss2q" Jul 23 01:43:11.227115 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.226956 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg2lh\" (UniqueName: \"kubernetes.io/projected/3f761efa-e6f1-4dbe-8702-0c2a1bab3730-kube-api-access-gg2lh\") pod \"maas-controller-cc8bb7484-qss2q\" (UID: \"3f761efa-e6f1-4dbe-8702-0c2a1bab3730\") " pod="opendatahub/maas-controller-cc8bb7484-qss2q" Jul 23 01:43:11.327433 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.327396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f761efa-e6f1-4dbe-8702-0c2a1bab3730-webhook-cert\") pod \"maas-controller-cc8bb7484-qss2q\" (UID: \"3f761efa-e6f1-4dbe-8702-0c2a1bab3730\") " pod="opendatahub/maas-controller-cc8bb7484-qss2q" Jul 23 01:43:11.327675 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.327451 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg2lh\" (UniqueName: \"kubernetes.io/projected/3f761efa-e6f1-4dbe-8702-0c2a1bab3730-kube-api-access-gg2lh\") pod \"maas-controller-cc8bb7484-qss2q\" (UID: \"3f761efa-e6f1-4dbe-8702-0c2a1bab3730\") " pod="opendatahub/maas-controller-cc8bb7484-qss2q" Jul 23 01:43:11.329901 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.329875 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f761efa-e6f1-4dbe-8702-0c2a1bab3730-webhook-cert\") pod \"maas-controller-cc8bb7484-qss2q\" (UID: \"3f761efa-e6f1-4dbe-8702-0c2a1bab3730\") " pod="opendatahub/maas-controller-cc8bb7484-qss2q" Jul 23 01:43:11.335556 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.335533 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg2lh\" (UniqueName: \"kubernetes.io/projected/3f761efa-e6f1-4dbe-8702-0c2a1bab3730-kube-api-access-gg2lh\") pod \"maas-controller-cc8bb7484-qss2q\" (UID: \"3f761efa-e6f1-4dbe-8702-0c2a1bab3730\") " pod="opendatahub/maas-controller-cc8bb7484-qss2q" Jul 23 01:43:11.412827 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.412744 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc8bb7484-qss2q" Jul 23 01:43:11.546755 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.546722 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc8bb7484-qss2q"] Jul 23 01:43:11.548194 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:43:11.548171 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f761efa_e6f1_4dbe_8702_0c2a1bab3730.slice/crio-52dc01c313afd816cce1857dffe59eee10089e2f6df2b1eedc964400b7afbe9e WatchSource:0}: Error finding container 52dc01c313afd816cce1857dffe59eee10089e2f6df2b1eedc964400b7afbe9e: Status 404 returned error can't find the container with id 52dc01c313afd816cce1857dffe59eee10089e2f6df2b1eedc964400b7afbe9e Jul 23 01:43:11.976614 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:11.976586 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2866217e-ac60-4435-b023-cc1b3cddc6a9" path="/var/lib/kubelet/pods/2866217e-ac60-4435-b023-cc1b3cddc6a9/volumes" Jul 23 01:43:12.316004 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:12.315903 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc8bb7484-qss2q" event={"ID":"3f761efa-e6f1-4dbe-8702-0c2a1bab3730","Type":"ContainerStarted","Data":"f3283ca6921b3f75f1d0de35d6e5b5b29a32707c5af03c81d793f08847902caa"} Jul 23 01:43:12.316004 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:12.315967 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cc8bb7484-qss2q" Jul 23 01:43:12.316004 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:12.315985 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc8bb7484-qss2q" event={"ID":"3f761efa-e6f1-4dbe-8702-0c2a1bab3730","Type":"ContainerStarted","Data":"52dc01c313afd816cce1857dffe59eee10089e2f6df2b1eedc964400b7afbe9e"} Jul 23 01:43:12.331969 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:12.331921 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cc8bb7484-qss2q" podStartSLOduration=0.925360452 podStartE2EDuration="1.331906219s" totalImagesPullingTime="406.545767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:43:12.330065992 +0000 UTC m=+904.875015169" watchObservedRunningTime="2026-07-23 01:43:12.331906219 +0000 UTC m=+904.876855409" Jul 23 01:43:23.324569 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:43:23.324535 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cc8bb7484-qss2q" Jul 23 01:44:07.986894 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:44:07.986864 2572 scope.go:117] "RemoveContainer" containerID="530b05d5c9986a78ac3ff28f9ff54d55e26d1802809bd2bca1bae4f2adf456e4" Jul 23 01:45:00.141742 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:00.141708 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746185-8fvcc"] Jul 23 01:45:00.145215 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:00.145199 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746185-8fvcc" Jul 23 01:45:00.147445 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:00.147419 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 01:45:00.154705 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:00.154690 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746185-8fvcc" Jul 23 01:45:00.158826 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:00.158802 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746185-8fvcc"] Jul 23 01:45:00.289605 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:00.289578 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746185-8fvcc"] Jul 23 01:45:00.291496 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:45:00.291473 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2cafea_5819_45e3_b183_fba8ac187c93.slice/crio-b46abccc22283cb9bed4ca559b1add92a4fdde4d57e58c8e25e3e1f14daa099f WatchSource:0}: Error finding container b46abccc22283cb9bed4ca559b1add92a4fdde4d57e58c8e25e3e1f14daa099f: Status 404 returned error can't find the container with id b46abccc22283cb9bed4ca559b1add92a4fdde4d57e58c8e25e3e1f14daa099f Jul 23 01:45:00.293845 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:00.293828 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 01:45:00.861584 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:00.861546 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746185-8fvcc" event={"ID":"6d2cafea-5819-45e3-b183-fba8ac187c93","Type":"ContainerStarted","Data":"b46abccc22283cb9bed4ca559b1add92a4fdde4d57e58c8e25e3e1f14daa099f"} Jul 23 01:45:03.878219 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:03.878173 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746185-8fvcc" event={"ID":"6d2cafea-5819-45e3-b183-fba8ac187c93","Type":"ContainerStarted","Data":"ca63a267364bab86444f9a37620838b0de199261cfd941b650004a5058ab8cd8"} Jul 23 01:45:03.894972 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:03.894863 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746185-8fvcc" podStartSLOduration=0.816693634 podStartE2EDuration="3.89484254s" totalImagesPullingTime="3.078148906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:45:03.892917164 +0000 UTC m=+1016.437866338" watchObservedRunningTime="2026-07-23 01:45:03.89484254 +0000 UTC m=+1016.439791712" Jul 23 01:45:04.885747 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:04.885707 2572 generic.go:356] "Generic (PLEG): container finished" podID="6d2cafea-5819-45e3-b183-fba8ac187c93" containerID="ca63a267364bab86444f9a37620838b0de199261cfd941b650004a5058ab8cd8" exitCode=0 Jul 23 01:45:04.886124 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:04.885796 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746185-8fvcc" event={"ID":"6d2cafea-5819-45e3-b183-fba8ac187c93","Type":"ContainerDied","Data":"ca63a267364bab86444f9a37620838b0de199261cfd941b650004a5058ab8cd8"} Jul 23 01:45:06.024107 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:06.024080 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746185-8fvcc" Jul 23 01:45:06.898127 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:06.898089 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746185-8fvcc" event={"ID":"6d2cafea-5819-45e3-b183-fba8ac187c93","Type":"ContainerDied","Data":"b46abccc22283cb9bed4ca559b1add92a4fdde4d57e58c8e25e3e1f14daa099f"} Jul 23 01:45:06.898127 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:06.898108 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746185-8fvcc" Jul 23 01:45:06.898392 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:45:06.898139 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b46abccc22283cb9bed4ca559b1add92a4fdde4d57e58c8e25e3e1f14daa099f" Jul 23 01:56:07.061257 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:07.061225 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz"] Jul 23 01:56:07.061692 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:07.061605 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2cafea-5819-45e3-b183-fba8ac187c93" containerName="cleanup" Jul 23 01:56:07.061692 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:07.061616 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2cafea-5819-45e3-b183-fba8ac187c93" containerName="cleanup" Jul 23 01:56:07.061768 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:07.061698 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d2cafea-5819-45e3-b183-fba8ac187c93" containerName="cleanup" Jul 23 01:56:07.064775 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:07.064747 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz" Jul 23 01:56:07.067334 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:07.067311 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 01:56:07.071819 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:07.071782 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz"] Jul 23 01:56:07.178737 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:07.178707 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc21303c-20ca-45ec-afaa-e6b2dfe627de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-338263ab-7z6lz\" (UID: \"cc21303c-20ca-45ec-afaa-e6b2dfe627de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz" Jul 23 01:56:07.279748 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:07.279708 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc21303c-20ca-45ec-afaa-e6b2dfe627de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-338263ab-7z6lz\" (UID: \"cc21303c-20ca-45ec-afaa-e6b2dfe627de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz" Jul 23 01:56:07.280324 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:07.280304 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc21303c-20ca-45ec-afaa-e6b2dfe627de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-338263ab-7z6lz\" (UID: \"cc21303c-20ca-45ec-afaa-e6b2dfe627de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz" Jul 23 01:56:07.377341 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:07.377297 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz" Jul 23 01:56:07.513931 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:07.513901 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz"] Jul 23 01:56:07.515411 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:56:07.515386 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc21303c_20ca_45ec_afaa_e6b2dfe627de.slice/crio-1d4202b763e6503af0fea2e3d8dbc9b52f2fcaa5e47d124b5268c07542901610 WatchSource:0}: Error finding container 1d4202b763e6503af0fea2e3d8dbc9b52f2fcaa5e47d124b5268c07542901610: Status 404 returned error can't find the container with id 1d4202b763e6503af0fea2e3d8dbc9b52f2fcaa5e47d124b5268c07542901610 Jul 23 01:56:08.285865 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:08.285824 2572 generic.go:356] "Generic (PLEG): container finished" podID="cc21303c-20ca-45ec-afaa-e6b2dfe627de" containerID="45f535a42b95e11180a68bdde5a81fcdaf176ea39f1e9374a539d1ca3f948984" exitCode=0 Jul 23 01:56:08.286249 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:08.285911 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz" event={"ID":"cc21303c-20ca-45ec-afaa-e6b2dfe627de","Type":"ContainerDied","Data":"45f535a42b95e11180a68bdde5a81fcdaf176ea39f1e9374a539d1ca3f948984"} Jul 23 01:56:08.286249 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:08.285969 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz" event={"ID":"cc21303c-20ca-45ec-afaa-e6b2dfe627de","Type":"ContainerStarted","Data":"1d4202b763e6503af0fea2e3d8dbc9b52f2fcaa5e47d124b5268c07542901610"} Jul 23 01:56:09.426798 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:09.426774 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz" Jul 23 01:56:09.599775 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:09.599747 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc21303c-20ca-45ec-afaa-e6b2dfe627de-maas-api-service-ca\") pod \"cc21303c-20ca-45ec-afaa-e6b2dfe627de\" (UID: \"cc21303c-20ca-45ec-afaa-e6b2dfe627de\") " Jul 23 01:56:09.600149 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:09.600128 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc21303c-20ca-45ec-afaa-e6b2dfe627de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc21303c-20ca-45ec-afaa-e6b2dfe627de" (UID: "cc21303c-20ca-45ec-afaa-e6b2dfe627de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:56:09.700680 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:09.700623 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc21303c-20ca-45ec-afaa-e6b2dfe627de-maas-api-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:56:10.297904 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:10.297818 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz" Jul 23 01:56:10.297904 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:10.297855 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz" event={"ID":"cc21303c-20ca-45ec-afaa-e6b2dfe627de","Type":"ContainerDied","Data":"1d4202b763e6503af0fea2e3d8dbc9b52f2fcaa5e47d124b5268c07542901610"} Jul 23 01:56:10.298094 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:10.297910 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d4202b763e6503af0fea2e3d8dbc9b52f2fcaa5e47d124b5268c07542901610" Jul 23 01:56:22.150428 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:22.150388 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz"] Jul 23 01:56:22.152456 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:22.152436 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-338263ab-7z6lz"] Jul 23 01:56:23.975906 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:23.975874 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc21303c-20ca-45ec-afaa-e6b2dfe627de" path="/var/lib/kubelet/pods/cc21303c-20ca-45ec-afaa-e6b2dfe627de/volumes" Jul 23 01:56:50.810624 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:50.810588 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k"] Jul 23 01:56:50.811102 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:50.810984 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc21303c-20ca-45ec-afaa-e6b2dfe627de" containerName="revoke-keys" Jul 23 01:56:50.811102 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:50.810998 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc21303c-20ca-45ec-afaa-e6b2dfe627de" containerName="revoke-keys" Jul 23 01:56:50.811102 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:50.811073 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc21303c-20ca-45ec-afaa-e6b2dfe627de" containerName="revoke-keys" Jul 23 01:56:50.813827 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:50.813806 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k" Jul 23 01:56:50.815957 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:50.815941 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 01:56:50.820336 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:50.820309 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k"] Jul 23 01:56:50.965478 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:50.965443 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1378c9f1-71d6-41af-aded-a8b95387d541-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k\" (UID: \"1378c9f1-71d6-41af-aded-a8b95387d541\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k" Jul 23 01:56:51.066970 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:51.066886 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1378c9f1-71d6-41af-aded-a8b95387d541-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k\" (UID: \"1378c9f1-71d6-41af-aded-a8b95387d541\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k" Jul 23 01:56:51.067479 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:51.067460 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1378c9f1-71d6-41af-aded-a8b95387d541-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k\" (UID: \"1378c9f1-71d6-41af-aded-a8b95387d541\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k" Jul 23 01:56:51.127333 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:51.127302 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k" Jul 23 01:56:51.469750 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:51.469714 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k"] Jul 23 01:56:51.472048 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:56:51.472016 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1378c9f1_71d6_41af_aded_a8b95387d541.slice/crio-408b9ced38e10bb8bfe75c0e880e982c0adad52ce6118876d7014bb76759a601 WatchSource:0}: Error finding container 408b9ced38e10bb8bfe75c0e880e982c0adad52ce6118876d7014bb76759a601: Status 404 returned error can't find the container with id 408b9ced38e10bb8bfe75c0e880e982c0adad52ce6118876d7014bb76759a601 Jul 23 01:56:51.518221 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:51.518193 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k" event={"ID":"1378c9f1-71d6-41af-aded-a8b95387d541","Type":"ContainerStarted","Data":"408b9ced38e10bb8bfe75c0e880e982c0adad52ce6118876d7014bb76759a601"} Jul 23 01:56:52.525223 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:52.525127 2572 generic.go:356] "Generic (PLEG): container finished" podID="1378c9f1-71d6-41af-aded-a8b95387d541" containerID="1e19553d418ca176f6661d6a47a34ccf8f0657c3a2ab7ef51ef883708605264f" exitCode=0 Jul 23 01:56:52.525623 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:52.525216 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k" event={"ID":"1378c9f1-71d6-41af-aded-a8b95387d541","Type":"ContainerDied","Data":"1e19553d418ca176f6661d6a47a34ccf8f0657c3a2ab7ef51ef883708605264f"} Jul 23 01:56:53.668698 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:53.668671 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k" Jul 23 01:56:53.794513 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:53.794408 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1378c9f1-71d6-41af-aded-a8b95387d541-maas-api-service-ca\") pod \"1378c9f1-71d6-41af-aded-a8b95387d541\" (UID: \"1378c9f1-71d6-41af-aded-a8b95387d541\") " Jul 23 01:56:53.794871 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:53.794839 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1378c9f1-71d6-41af-aded-a8b95387d541-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1378c9f1-71d6-41af-aded-a8b95387d541" (UID: "1378c9f1-71d6-41af-aded-a8b95387d541"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:56:53.896071 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:53.896032 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1378c9f1-71d6-41af-aded-a8b95387d541-maas-api-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:56:54.539021 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:54.538992 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k" Jul 23 01:56:54.539021 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:54.538998 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k" event={"ID":"1378c9f1-71d6-41af-aded-a8b95387d541","Type":"ContainerDied","Data":"408b9ced38e10bb8bfe75c0e880e982c0adad52ce6118876d7014bb76759a601"} Jul 23 01:56:54.539392 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:56:54.539046 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="408b9ced38e10bb8bfe75c0e880e982c0adad52ce6118876d7014bb76759a601" Jul 23 01:57:05.886692 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:05.886657 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k"] Jul 23 01:57:05.889625 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:05.889599 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dafa5c71-2v85k"] Jul 23 01:57:05.975679 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:05.975626 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1378c9f1-71d6-41af-aded-a8b95387d541" path="/var/lib/kubelet/pods/1378c9f1-71d6-41af-aded-a8b95387d541/volumes" Jul 23 01:57:34.739076 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:34.739045 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj"] Jul 23 01:57:34.739817 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:34.739541 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1378c9f1-71d6-41af-aded-a8b95387d541" containerName="revoke-keys" Jul 23 01:57:34.739817 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:34.739561 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1378c9f1-71d6-41af-aded-a8b95387d541" containerName="revoke-keys" Jul 23 01:57:34.739817 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:34.739672 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1378c9f1-71d6-41af-aded-a8b95387d541" containerName="revoke-keys" Jul 23 01:57:34.742957 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:34.742934 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj" Jul 23 01:57:34.745345 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:34.745322 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 01:57:34.749187 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:34.749166 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj"] Jul 23 01:57:34.831254 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:34.831218 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6138a1-72f1-4bee-bd69-5447b3876537-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj\" (UID: \"ef6138a1-72f1-4bee-bd69-5447b3876537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj" Jul 23 01:57:34.932622 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:34.932584 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6138a1-72f1-4bee-bd69-5447b3876537-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj\" (UID: \"ef6138a1-72f1-4bee-bd69-5447b3876537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj" Jul 23 01:57:34.933231 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:34.933203 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6138a1-72f1-4bee-bd69-5447b3876537-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj\" (UID: \"ef6138a1-72f1-4bee-bd69-5447b3876537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj" Jul 23 01:57:35.055089 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:35.055002 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj" Jul 23 01:57:35.192676 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:35.192647 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj"] Jul 23 01:57:35.194627 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:57:35.194599 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6138a1_72f1_4bee_bd69_5447b3876537.slice/crio-b7d84da92a19d6d3128c42da5a97b551a88c76819447702ee4672b6f587dffa5 WatchSource:0}: Error finding container b7d84da92a19d6d3128c42da5a97b551a88c76819447702ee4672b6f587dffa5: Status 404 returned error can't find the container with id b7d84da92a19d6d3128c42da5a97b551a88c76819447702ee4672b6f587dffa5 Jul 23 01:57:35.750825 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:35.750719 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj" event={"ID":"ef6138a1-72f1-4bee-bd69-5447b3876537","Type":"ContainerStarted","Data":"8b773a5e9509157b30681e19106dedcd52835b2c5e8a44f45e967193a7a7fd32"} Jul 23 01:57:35.750825 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:35.750773 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj" event={"ID":"ef6138a1-72f1-4bee-bd69-5447b3876537","Type":"ContainerStarted","Data":"b7d84da92a19d6d3128c42da5a97b551a88c76819447702ee4672b6f587dffa5"} Jul 23 01:57:35.767262 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:35.767211 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj" podStartSLOduration=1.767195729 podStartE2EDuration="1.767195729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:57:35.766381344 +0000 UTC m=+1768.311330536" watchObservedRunningTime="2026-07-23 01:57:35.767195729 +0000 UTC m=+1768.312144902" Jul 23 01:57:36.757326 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:36.757294 2572 generic.go:356] "Generic (PLEG): container finished" podID="ef6138a1-72f1-4bee-bd69-5447b3876537" containerID="8b773a5e9509157b30681e19106dedcd52835b2c5e8a44f45e967193a7a7fd32" exitCode=0 Jul 23 01:57:36.757726 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:36.757352 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj" event={"ID":"ef6138a1-72f1-4bee-bd69-5447b3876537","Type":"ContainerDied","Data":"8b773a5e9509157b30681e19106dedcd52835b2c5e8a44f45e967193a7a7fd32"} Jul 23 01:57:37.891010 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:37.890983 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj" Jul 23 01:57:37.957778 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:37.957745 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6138a1-72f1-4bee-bd69-5447b3876537-maas-api-service-ca\") pod \"ef6138a1-72f1-4bee-bd69-5447b3876537\" (UID: \"ef6138a1-72f1-4bee-bd69-5447b3876537\") " Jul 23 01:57:37.958082 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:37.958057 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6138a1-72f1-4bee-bd69-5447b3876537-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef6138a1-72f1-4bee-bd69-5447b3876537" (UID: "ef6138a1-72f1-4bee-bd69-5447b3876537"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:57:38.059162 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:38.059071 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6138a1-72f1-4bee-bd69-5447b3876537-maas-api-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:57:38.769302 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:38.769274 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj" Jul 23 01:57:38.769468 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:38.769321 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj" event={"ID":"ef6138a1-72f1-4bee-bd69-5447b3876537","Type":"ContainerDied","Data":"b7d84da92a19d6d3128c42da5a97b551a88c76819447702ee4672b6f587dffa5"} Jul 23 01:57:38.769468 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:38.769368 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7d84da92a19d6d3128c42da5a97b551a88c76819447702ee4672b6f587dffa5" Jul 23 01:57:49.821987 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:49.821948 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj"] Jul 23 01:57:49.826299 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:49.826271 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bc37934-p9xcj"] Jul 23 01:57:49.975102 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:57:49.975069 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef6138a1-72f1-4bee-bd69-5447b3876537" path="/var/lib/kubelet/pods/ef6138a1-72f1-4bee-bd69-5447b3876537/volumes" Jul 23 01:58:19.003146 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:19.003108 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd"] Jul 23 01:58:19.003739 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:19.003719 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef6138a1-72f1-4bee-bd69-5447b3876537" containerName="revoke-keys" Jul 23 01:58:19.003822 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:19.003741 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef6138a1-72f1-4bee-bd69-5447b3876537" containerName="revoke-keys" Jul 23 01:58:19.003877 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:19.003854 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef6138a1-72f1-4bee-bd69-5447b3876537" containerName="revoke-keys" Jul 23 01:58:19.006816 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:19.006790 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd" Jul 23 01:58:19.009097 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:19.009074 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 01:58:19.014050 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:19.014014 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd"] Jul 23 01:58:19.119888 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:19.119856 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2d5a64-0033-40c9-b4cb-2755cd2c9a04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6955f559-lk6vd\" (UID: \"7a2d5a64-0033-40c9-b4cb-2755cd2c9a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd" Jul 23 01:58:19.221222 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:19.221180 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2d5a64-0033-40c9-b4cb-2755cd2c9a04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6955f559-lk6vd\" (UID: \"7a2d5a64-0033-40c9-b4cb-2755cd2c9a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd" Jul 23 01:58:19.221876 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:19.221857 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2d5a64-0033-40c9-b4cb-2755cd2c9a04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6955f559-lk6vd\" (UID: \"7a2d5a64-0033-40c9-b4cb-2755cd2c9a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd" Jul 23 01:58:19.320897 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:19.320806 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd" Jul 23 01:58:19.461074 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:19.461049 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd"] Jul 23 01:58:19.462569 ip-10-0-131-117 kubenswrapper[2572]: W0723 01:58:19.462530 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2d5a64_0033_40c9_b4cb_2755cd2c9a04.slice/crio-0faea910a995ac9167e84cca3404a2e03ddac4c68be848f05ced40d1263ca16f WatchSource:0}: Error finding container 0faea910a995ac9167e84cca3404a2e03ddac4c68be848f05ced40d1263ca16f: Status 404 returned error can't find the container with id 0faea910a995ac9167e84cca3404a2e03ddac4c68be848f05ced40d1263ca16f Jul 23 01:58:20.003931 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:20.003822 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd" event={"ID":"7a2d5a64-0033-40c9-b4cb-2755cd2c9a04","Type":"ContainerStarted","Data":"4e9254fc211e5881f439e805b878763c0a71cbb2e618dcc5662fb25039ebd8c2"} Jul 23 01:58:20.003931 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:20.003894 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd" event={"ID":"7a2d5a64-0033-40c9-b4cb-2755cd2c9a04","Type":"ContainerStarted","Data":"0faea910a995ac9167e84cca3404a2e03ddac4c68be848f05ced40d1263ca16f"} Jul 23 01:58:20.019737 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:20.019679 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd" podStartSLOduration=2.019662479 podStartE2EDuration="2.019662479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:58:20.017477448 +0000 UTC m=+1812.562426621" watchObservedRunningTime="2026-07-23 01:58:20.019662479 +0000 UTC m=+1812.564611651" Jul 23 01:58:21.010913 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:21.010881 2572 generic.go:356] "Generic (PLEG): container finished" podID="7a2d5a64-0033-40c9-b4cb-2755cd2c9a04" containerID="4e9254fc211e5881f439e805b878763c0a71cbb2e618dcc5662fb25039ebd8c2" exitCode=0 Jul 23 01:58:21.011310 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:21.010965 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd" event={"ID":"7a2d5a64-0033-40c9-b4cb-2755cd2c9a04","Type":"ContainerDied","Data":"4e9254fc211e5881f439e805b878763c0a71cbb2e618dcc5662fb25039ebd8c2"} Jul 23 01:58:22.150795 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:22.150768 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd" Jul 23 01:58:22.247236 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:22.247194 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2d5a64-0033-40c9-b4cb-2755cd2c9a04-maas-api-service-ca\") pod \"7a2d5a64-0033-40c9-b4cb-2755cd2c9a04\" (UID: \"7a2d5a64-0033-40c9-b4cb-2755cd2c9a04\") " Jul 23 01:58:22.247617 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:22.247590 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2d5a64-0033-40c9-b4cb-2755cd2c9a04-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7a2d5a64-0033-40c9-b4cb-2755cd2c9a04" (UID: "7a2d5a64-0033-40c9-b4cb-2755cd2c9a04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:58:22.347963 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:22.347929 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2d5a64-0033-40c9-b4cb-2755cd2c9a04-maas-api-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 01:58:23.022869 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:23.022835 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd" Jul 23 01:58:23.023026 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:23.022832 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd" event={"ID":"7a2d5a64-0033-40c9-b4cb-2755cd2c9a04","Type":"ContainerDied","Data":"0faea910a995ac9167e84cca3404a2e03ddac4c68be848f05ced40d1263ca16f"} Jul 23 01:58:23.023026 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:23.022961 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0faea910a995ac9167e84cca3404a2e03ddac4c68be848f05ced40d1263ca16f" Jul 23 01:58:34.086437 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:34.086397 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd"] Jul 23 01:58:34.090389 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:34.090364 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6955f559-lk6vd"] Jul 23 01:58:35.976260 ip-10-0-131-117 kubenswrapper[2572]: I0723 01:58:35.976227 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a2d5a64-0033-40c9-b4cb-2755cd2c9a04" path="/var/lib/kubelet/pods/7a2d5a64-0033-40c9-b4cb-2755cd2c9a04/volumes" Jul 23 02:00:25.235552 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:25.235517 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt"] Jul 23 02:00:25.236041 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:25.235922 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2d5a64-0033-40c9-b4cb-2755cd2c9a04" containerName="revoke-keys" Jul 23 02:00:25.236041 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:25.235936 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2d5a64-0033-40c9-b4cb-2755cd2c9a04" containerName="revoke-keys" Jul 23 02:00:25.236041 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:25.236012 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2d5a64-0033-40c9-b4cb-2755cd2c9a04" containerName="revoke-keys" Jul 23 02:00:25.239245 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:25.239224 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt" Jul 23 02:00:25.241794 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:25.241755 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 02:00:25.245999 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:25.245963 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt"] Jul 23 02:00:25.318065 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:25.318028 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63e28ca-561e-4f55-ae53-b7a8d944c9c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt\" (UID: \"d63e28ca-561e-4f55-ae53-b7a8d944c9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt" Jul 23 02:00:25.418587 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:25.418549 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63e28ca-561e-4f55-ae53-b7a8d944c9c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt\" (UID: \"d63e28ca-561e-4f55-ae53-b7a8d944c9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt" Jul 23 02:00:25.419181 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:25.419161 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63e28ca-561e-4f55-ae53-b7a8d944c9c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt\" (UID: \"d63e28ca-561e-4f55-ae53-b7a8d944c9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt" Jul 23 02:00:25.551533 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:25.551452 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt" Jul 23 02:00:25.694377 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:25.694348 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt"] Jul 23 02:00:25.696256 ip-10-0-131-117 kubenswrapper[2572]: W0723 02:00:25.696228 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63e28ca_561e_4f55_ae53_b7a8d944c9c5.slice/crio-a729570cb2da6fef04240546ec85e3b04a11a5726adb95dd836a416c747aff0c WatchSource:0}: Error finding container a729570cb2da6fef04240546ec85e3b04a11a5726adb95dd836a416c747aff0c: Status 404 returned error can't find the container with id a729570cb2da6fef04240546ec85e3b04a11a5726adb95dd836a416c747aff0c Jul 23 02:00:26.680681 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:26.680621 2572 generic.go:356] "Generic (PLEG): container finished" podID="d63e28ca-561e-4f55-ae53-b7a8d944c9c5" containerID="6fac75e91c00d42b2494963a2d08dc4236f93e14547a8eb7217ab38e7eac5c53" exitCode=0 Jul 23 02:00:26.681070 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:26.680680 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt" event={"ID":"d63e28ca-561e-4f55-ae53-b7a8d944c9c5","Type":"ContainerDied","Data":"6fac75e91c00d42b2494963a2d08dc4236f93e14547a8eb7217ab38e7eac5c53"} Jul 23 02:00:26.681070 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:26.680735 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt" event={"ID":"d63e28ca-561e-4f55-ae53-b7a8d944c9c5","Type":"ContainerStarted","Data":"a729570cb2da6fef04240546ec85e3b04a11a5726adb95dd836a416c747aff0c"} Jul 23 02:00:27.819908 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:27.819878 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt" Jul 23 02:00:27.941509 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:27.941411 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63e28ca-561e-4f55-ae53-b7a8d944c9c5-maas-api-service-ca\") pod \"d63e28ca-561e-4f55-ae53-b7a8d944c9c5\" (UID: \"d63e28ca-561e-4f55-ae53-b7a8d944c9c5\") " Jul 23 02:00:27.941829 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:27.941805 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63e28ca-561e-4f55-ae53-b7a8d944c9c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d63e28ca-561e-4f55-ae53-b7a8d944c9c5" (UID: "d63e28ca-561e-4f55-ae53-b7a8d944c9c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 02:00:28.042899 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:28.042854 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63e28ca-561e-4f55-ae53-b7a8d944c9c5-maas-api-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:00:28.693325 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:28.693288 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt" Jul 23 02:00:28.693325 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:28.693293 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt" event={"ID":"d63e28ca-561e-4f55-ae53-b7a8d944c9c5","Type":"ContainerDied","Data":"a729570cb2da6fef04240546ec85e3b04a11a5726adb95dd836a416c747aff0c"} Jul 23 02:00:28.693545 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:28.693337 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a729570cb2da6fef04240546ec85e3b04a11a5726adb95dd836a416c747aff0c" Jul 23 02:00:40.312158 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:40.312122 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt"] Jul 23 02:00:40.314387 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:40.314360 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcbcd8ed-ffjgt"] Jul 23 02:00:41.975566 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:00:41.975532 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d63e28ca-561e-4f55-ae53-b7a8d944c9c5" path="/var/lib/kubelet/pods/d63e28ca-561e-4f55-ae53-b7a8d944c9c5/volumes" Jul 23 02:01:41.292998 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:41.292961 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt"] Jul 23 02:01:41.293527 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:41.293391 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d63e28ca-561e-4f55-ae53-b7a8d944c9c5" containerName="revoke-keys" Jul 23 02:01:41.293527 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:41.293409 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d63e28ca-561e-4f55-ae53-b7a8d944c9c5" containerName="revoke-keys" Jul 23 02:01:41.293527 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:41.293479 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d63e28ca-561e-4f55-ae53-b7a8d944c9c5" containerName="revoke-keys" Jul 23 02:01:41.296601 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:41.296581 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt" Jul 23 02:01:41.299030 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:41.299009 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 02:01:41.304859 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:41.304835 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt"] Jul 23 02:01:41.459373 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:41.459334 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4208d16-5c2d-4024-8df3-169bf465160b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d207b89-jntjt\" (UID: \"c4208d16-5c2d-4024-8df3-169bf465160b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt" Jul 23 02:01:41.560404 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:41.560311 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4208d16-5c2d-4024-8df3-169bf465160b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d207b89-jntjt\" (UID: \"c4208d16-5c2d-4024-8df3-169bf465160b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt" Jul 23 02:01:41.560962 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:41.560941 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4208d16-5c2d-4024-8df3-169bf465160b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d207b89-jntjt\" (UID: \"c4208d16-5c2d-4024-8df3-169bf465160b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt" Jul 23 02:01:41.608382 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:41.608348 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt" Jul 23 02:01:41.745423 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:41.745387 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt"] Jul 23 02:01:41.747181 ip-10-0-131-117 kubenswrapper[2572]: W0723 02:01:41.747152 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4208d16_5c2d_4024_8df3_169bf465160b.slice/crio-c2080e5ed1902860fb0b9272a8bf0f5f59be86e8fc635ca733c54ae10a85ff3a WatchSource:0}: Error finding container c2080e5ed1902860fb0b9272a8bf0f5f59be86e8fc635ca733c54ae10a85ff3a: Status 404 returned error can't find the container with id c2080e5ed1902860fb0b9272a8bf0f5f59be86e8fc635ca733c54ae10a85ff3a Jul 23 02:01:42.103272 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:42.103229 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt" event={"ID":"c4208d16-5c2d-4024-8df3-169bf465160b","Type":"ContainerStarted","Data":"9771b38326858b907aa50bd1c5ad5a6010c2ba3ac09dd0abda760cb11819a6b3"} Jul 23 02:01:42.103437 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:42.103296 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt" event={"ID":"c4208d16-5c2d-4024-8df3-169bf465160b","Type":"ContainerStarted","Data":"c2080e5ed1902860fb0b9272a8bf0f5f59be86e8fc635ca733c54ae10a85ff3a"} Jul 23 02:01:42.120175 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:42.120126 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt" podStartSLOduration=1.120109797 podStartE2EDuration="1.120109797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:01:42.11880017 +0000 UTC m=+2014.663749343" watchObservedRunningTime="2026-07-23 02:01:42.120109797 +0000 UTC m=+2014.665059038" Jul 23 02:01:43.110137 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:43.110104 2572 generic.go:356] "Generic (PLEG): container finished" podID="c4208d16-5c2d-4024-8df3-169bf465160b" containerID="9771b38326858b907aa50bd1c5ad5a6010c2ba3ac09dd0abda760cb11819a6b3" exitCode=0 Jul 23 02:01:43.110545 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:43.110196 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt" event={"ID":"c4208d16-5c2d-4024-8df3-169bf465160b","Type":"ContainerDied","Data":"9771b38326858b907aa50bd1c5ad5a6010c2ba3ac09dd0abda760cb11819a6b3"} Jul 23 02:01:44.244754 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:44.244730 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt" Jul 23 02:01:44.387652 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:44.387543 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4208d16-5c2d-4024-8df3-169bf465160b-maas-api-service-ca\") pod \"c4208d16-5c2d-4024-8df3-169bf465160b\" (UID: \"c4208d16-5c2d-4024-8df3-169bf465160b\") " Jul 23 02:01:44.387965 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:44.387943 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4208d16-5c2d-4024-8df3-169bf465160b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4208d16-5c2d-4024-8df3-169bf465160b" (UID: "c4208d16-5c2d-4024-8df3-169bf465160b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 02:01:44.488989 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:44.488953 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4208d16-5c2d-4024-8df3-169bf465160b-maas-api-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:01:45.124575 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:45.124540 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt" event={"ID":"c4208d16-5c2d-4024-8df3-169bf465160b","Type":"ContainerDied","Data":"c2080e5ed1902860fb0b9272a8bf0f5f59be86e8fc635ca733c54ae10a85ff3a"} Jul 23 02:01:45.124575 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:45.124558 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt" Jul 23 02:01:45.124820 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:01:45.124588 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2080e5ed1902860fb0b9272a8bf0f5f59be86e8fc635ca733c54ae10a85ff3a" Jul 23 02:02:01.378057 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:02:01.378021 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt"] Jul 23 02:02:01.380216 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:02:01.380193 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d207b89-jntjt"] Jul 23 02:02:01.975907 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:02:01.975877 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4208d16-5c2d-4024-8df3-169bf465160b" path="/var/lib/kubelet/pods/c4208d16-5c2d-4024-8df3-169bf465160b/volumes" Jul 23 02:02:08.159145 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:02:08.159107 2572 scope.go:117] "RemoveContainer" containerID="45f535a42b95e11180a68bdde5a81fcdaf176ea39f1e9374a539d1ca3f948984" Jul 23 02:03:08.181140 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:03:08.181107 2572 scope.go:117] "RemoveContainer" containerID="1e19553d418ca176f6661d6a47a34ccf8f0657c3a2ab7ef51ef883708605264f" Jul 23 02:04:08.201042 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:04:08.200963 2572 scope.go:117] "RemoveContainer" containerID="8b773a5e9509157b30681e19106dedcd52835b2c5e8a44f45e967193a7a7fd32" Jul 23 02:05:08.223911 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:08.223880 2572 scope.go:117] "RemoveContainer" containerID="4e9254fc211e5881f439e805b878763c0a71cbb2e618dcc5662fb25039ebd8c2" Jul 23 02:05:17.405704 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:17.405668 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v"] Jul 23 02:05:17.406166 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:17.406045 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4208d16-5c2d-4024-8df3-169bf465160b" containerName="revoke-keys" Jul 23 02:05:17.406166 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:17.406057 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4208d16-5c2d-4024-8df3-169bf465160b" containerName="revoke-keys" Jul 23 02:05:17.406166 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:17.406122 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4208d16-5c2d-4024-8df3-169bf465160b" containerName="revoke-keys" Jul 23 02:05:17.409275 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:17.409255 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v" Jul 23 02:05:17.411450 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:17.411427 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 02:05:17.415377 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:17.415353 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v"] Jul 23 02:05:17.534304 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:17.534266 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab0777-d5e2-439d-9892-927fdae8d0b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v\" (UID: \"6bab0777-d5e2-439d-9892-927fdae8d0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v" Jul 23 02:05:17.635743 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:17.635708 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab0777-d5e2-439d-9892-927fdae8d0b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v\" (UID: \"6bab0777-d5e2-439d-9892-927fdae8d0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v" Jul 23 02:05:17.636379 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:17.636361 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab0777-d5e2-439d-9892-927fdae8d0b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v\" (UID: \"6bab0777-d5e2-439d-9892-927fdae8d0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v" Jul 23 02:05:17.722360 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:17.722261 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v" Jul 23 02:05:17.855680 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:17.855621 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v"] Jul 23 02:05:17.857278 ip-10-0-131-117 kubenswrapper[2572]: W0723 02:05:17.857249 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bab0777_d5e2_439d_9892_927fdae8d0b4.slice/crio-ef9b9fc22782236e94ce54f2479547dcb023f9a62bd119a461f0c865819aee0f WatchSource:0}: Error finding container ef9b9fc22782236e94ce54f2479547dcb023f9a62bd119a461f0c865819aee0f: Status 404 returned error can't find the container with id ef9b9fc22782236e94ce54f2479547dcb023f9a62bd119a461f0c865819aee0f Jul 23 02:05:18.276708 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:18.276662 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v" event={"ID":"6bab0777-d5e2-439d-9892-927fdae8d0b4","Type":"ContainerStarted","Data":"a360378fc3e94dce10dffa01616f394083e9951e73eb39c4009c39611e1ea192"} Jul 23 02:05:18.276892 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:18.276750 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v" event={"ID":"6bab0777-d5e2-439d-9892-927fdae8d0b4","Type":"ContainerStarted","Data":"ef9b9fc22782236e94ce54f2479547dcb023f9a62bd119a461f0c865819aee0f"} Jul 23 02:05:18.291882 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:18.291835 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v" podStartSLOduration=1.291823113 podStartE2EDuration="1.291823113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:05:18.291331749 +0000 UTC m=+2230.836280922" watchObservedRunningTime="2026-07-23 02:05:18.291823113 +0000 UTC m=+2230.836772321" Jul 23 02:05:19.283348 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:19.283311 2572 generic.go:356] "Generic (PLEG): container finished" podID="6bab0777-d5e2-439d-9892-927fdae8d0b4" containerID="a360378fc3e94dce10dffa01616f394083e9951e73eb39c4009c39611e1ea192" exitCode=0 Jul 23 02:05:19.283831 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:19.283400 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v" event={"ID":"6bab0777-d5e2-439d-9892-927fdae8d0b4","Type":"ContainerDied","Data":"a360378fc3e94dce10dffa01616f394083e9951e73eb39c4009c39611e1ea192"} Jul 23 02:05:20.422384 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:20.422354 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v" Jul 23 02:05:20.558622 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:20.558512 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab0777-d5e2-439d-9892-927fdae8d0b4-maas-api-service-ca\") pod \"6bab0777-d5e2-439d-9892-927fdae8d0b4\" (UID: \"6bab0777-d5e2-439d-9892-927fdae8d0b4\") " Jul 23 02:05:20.558929 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:20.558907 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bab0777-d5e2-439d-9892-927fdae8d0b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6bab0777-d5e2-439d-9892-927fdae8d0b4" (UID: "6bab0777-d5e2-439d-9892-927fdae8d0b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 02:05:20.659498 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:20.659453 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab0777-d5e2-439d-9892-927fdae8d0b4-maas-api-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:05:21.294841 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:21.294808 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v" event={"ID":"6bab0777-d5e2-439d-9892-927fdae8d0b4","Type":"ContainerDied","Data":"ef9b9fc22782236e94ce54f2479547dcb023f9a62bd119a461f0c865819aee0f"} Jul 23 02:05:21.294841 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:21.294840 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v" Jul 23 02:05:21.295062 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:21.294868 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef9b9fc22782236e94ce54f2479547dcb023f9a62bd119a461f0c865819aee0f" Jul 23 02:05:32.498153 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:32.498066 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v"] Jul 23 02:05:32.501788 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:32.501759 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-99eab2-szx6v"] Jul 23 02:05:33.975429 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:33.975392 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bab0777-d5e2-439d-9892-927fdae8d0b4" path="/var/lib/kubelet/pods/6bab0777-d5e2-439d-9892-927fdae8d0b4/volumes" Jul 23 02:05:39.375991 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:39.375954 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp"] Jul 23 02:05:39.376471 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:39.376386 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bab0777-d5e2-439d-9892-927fdae8d0b4" containerName="revoke-keys" Jul 23 02:05:39.376471 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:39.376400 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bab0777-d5e2-439d-9892-927fdae8d0b4" containerName="revoke-keys" Jul 23 02:05:39.376471 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:39.376458 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bab0777-d5e2-439d-9892-927fdae8d0b4" containerName="revoke-keys" Jul 23 02:05:39.379788 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:39.379770 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp" Jul 23 02:05:39.382141 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:39.382118 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 02:05:39.386726 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:39.386698 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp"] Jul 23 02:05:39.529244 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:39.529207 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4254c30-68a4-4867-8c67-430c54e6c44c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-25d739-qtllp\" (UID: \"f4254c30-68a4-4867-8c67-430c54e6c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp" Jul 23 02:05:39.630851 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:39.630769 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4254c30-68a4-4867-8c67-430c54e6c44c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-25d739-qtllp\" (UID: \"f4254c30-68a4-4867-8c67-430c54e6c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp" Jul 23 02:05:39.631419 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:39.631394 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4254c30-68a4-4867-8c67-430c54e6c44c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-25d739-qtllp\" (UID: \"f4254c30-68a4-4867-8c67-430c54e6c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp" Jul 23 02:05:39.691773 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:39.691719 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp" Jul 23 02:05:40.034388 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:40.034358 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp"] Jul 23 02:05:40.036193 ip-10-0-131-117 kubenswrapper[2572]: W0723 02:05:40.036168 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4254c30_68a4_4867_8c67_430c54e6c44c.slice/crio-2f6aebfd57cfbf5cc75b59de0d770a7be76a2035ae6d9d135cec615c9c07ecec WatchSource:0}: Error finding container 2f6aebfd57cfbf5cc75b59de0d770a7be76a2035ae6d9d135cec615c9c07ecec: Status 404 returned error can't find the container with id 2f6aebfd57cfbf5cc75b59de0d770a7be76a2035ae6d9d135cec615c9c07ecec Jul 23 02:05:40.400517 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:40.400484 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp" event={"ID":"f4254c30-68a4-4867-8c67-430c54e6c44c","Type":"ContainerStarted","Data":"e6eebd5d1831d55e6353cf8612d84df5b2da16e113085d80bef52d17c1337c2d"} Jul 23 02:05:40.400999 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:40.400538 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp" event={"ID":"f4254c30-68a4-4867-8c67-430c54e6c44c","Type":"ContainerStarted","Data":"2f6aebfd57cfbf5cc75b59de0d770a7be76a2035ae6d9d135cec615c9c07ecec"} Jul 23 02:05:40.416651 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:40.416591 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp" podStartSLOduration=1.4165772730000001 podStartE2EDuration="1.416577273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:05:40.414052912 +0000 UTC m=+2252.959002103" watchObservedRunningTime="2026-07-23 02:05:40.416577273 +0000 UTC m=+2252.961526445" Jul 23 02:05:41.413853 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:41.413818 2572 generic.go:356] "Generic (PLEG): container finished" podID="f4254c30-68a4-4867-8c67-430c54e6c44c" containerID="e6eebd5d1831d55e6353cf8612d84df5b2da16e113085d80bef52d17c1337c2d" exitCode=0 Jul 23 02:05:41.414261 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:41.413905 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp" event={"ID":"f4254c30-68a4-4867-8c67-430c54e6c44c","Type":"ContainerDied","Data":"e6eebd5d1831d55e6353cf8612d84df5b2da16e113085d80bef52d17c1337c2d"} Jul 23 02:05:42.548450 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:42.548425 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp" Jul 23 02:05:42.657582 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:42.657529 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4254c30-68a4-4867-8c67-430c54e6c44c-maas-api-service-ca\") pod \"f4254c30-68a4-4867-8c67-430c54e6c44c\" (UID: \"f4254c30-68a4-4867-8c67-430c54e6c44c\") " Jul 23 02:05:42.657956 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:42.657930 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4254c30-68a4-4867-8c67-430c54e6c44c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4254c30-68a4-4867-8c67-430c54e6c44c" (UID: "f4254c30-68a4-4867-8c67-430c54e6c44c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 02:05:42.759232 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:42.759143 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4254c30-68a4-4867-8c67-430c54e6c44c-maas-api-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:05:43.425840 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:43.425797 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp" event={"ID":"f4254c30-68a4-4867-8c67-430c54e6c44c","Type":"ContainerDied","Data":"2f6aebfd57cfbf5cc75b59de0d770a7be76a2035ae6d9d135cec615c9c07ecec"} Jul 23 02:05:43.425840 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:43.425846 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f6aebfd57cfbf5cc75b59de0d770a7be76a2035ae6d9d135cec615c9c07ecec" Jul 23 02:05:43.426051 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:43.425814 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp" Jul 23 02:05:54.454716 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:54.454671 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp"] Jul 23 02:05:54.456836 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:54.456810 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-25d739-qtllp"] Jul 23 02:05:55.975496 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:05:55.975464 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4254c30-68a4-4867-8c67-430c54e6c44c" path="/var/lib/kubelet/pods/f4254c30-68a4-4867-8c67-430c54e6c44c/volumes" Jul 23 02:06:12.422765 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.422729 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8"] Jul 23 02:06:12.424946 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.423104 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4254c30-68a4-4867-8c67-430c54e6c44c" containerName="revoke-keys" Jul 23 02:06:12.424946 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.423116 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4254c30-68a4-4867-8c67-430c54e6c44c" containerName="revoke-keys" Jul 23 02:06:12.424946 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.423178 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4254c30-68a4-4867-8c67-430c54e6c44c" containerName="revoke-keys" Jul 23 02:06:12.426243 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.426228 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:12.428947 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.428924 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rzlfw\"" Jul 23 02:06:12.429473 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.429448 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cf5ce0-serving-cert\"" Jul 23 02:06:12.451397 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.451368 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8"] Jul 23 02:06:12.521510 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.521473 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/524cc1f0-9e2d-4b0f-a816-0172e326b49d-maas-api-tls\") pod \"maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8\" (UID: \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:12.521720 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.521522 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/524cc1f0-9e2d-4b0f-a816-0172e326b49d-openshift-service-ca\") pod \"maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8\" (UID: \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:12.521720 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.521654 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjvzw\" (UniqueName: \"kubernetes.io/projected/524cc1f0-9e2d-4b0f-a816-0172e326b49d-kube-api-access-sjvzw\") pod \"maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8\" (UID: \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:12.622694 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.622653 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/524cc1f0-9e2d-4b0f-a816-0172e326b49d-maas-api-tls\") pod \"maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8\" (UID: \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:12.622868 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.622708 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/524cc1f0-9e2d-4b0f-a816-0172e326b49d-openshift-service-ca\") pod \"maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8\" (UID: \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:12.622868 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.622782 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvzw\" (UniqueName: \"kubernetes.io/projected/524cc1f0-9e2d-4b0f-a816-0172e326b49d-kube-api-access-sjvzw\") pod \"maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8\" (UID: \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:12.623397 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.623372 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/524cc1f0-9e2d-4b0f-a816-0172e326b49d-openshift-service-ca\") pod \"maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8\" (UID: \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:12.625231 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.625206 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/524cc1f0-9e2d-4b0f-a816-0172e326b49d-maas-api-tls\") pod \"maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8\" (UID: \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:12.637290 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.637261 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvzw\" (UniqueName: \"kubernetes.io/projected/524cc1f0-9e2d-4b0f-a816-0172e326b49d-kube-api-access-sjvzw\") pod \"maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8\" (UID: \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:12.736921 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.736837 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:12.872428 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.872394 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8"] Jul 23 02:06:12.873495 ip-10-0-131-117 kubenswrapper[2572]: W0723 02:06:12.873470 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524cc1f0_9e2d_4b0f_a816_0172e326b49d.slice/crio-3ad279830367693d9fab378b3b81515227692f048e8ba13317363bd80fc77832 WatchSource:0}: Error finding container 3ad279830367693d9fab378b3b81515227692f048e8ba13317363bd80fc77832: Status 404 returned error can't find the container with id 3ad279830367693d9fab378b3b81515227692f048e8ba13317363bd80fc77832 Jul 23 02:06:12.874864 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:12.874844 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 02:06:13.593716 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:13.593673 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" event={"ID":"524cc1f0-9e2d-4b0f-a816-0172e326b49d","Type":"ContainerStarted","Data":"3ad279830367693d9fab378b3b81515227692f048e8ba13317363bd80fc77832"} Jul 23 02:06:17.616504 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:17.616465 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" event={"ID":"524cc1f0-9e2d-4b0f-a816-0172e326b49d","Type":"ContainerStarted","Data":"0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab"} Jul 23 02:06:17.616954 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:17.616533 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:17.634566 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:17.634513 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" podStartSLOduration=1.8181726409999999 podStartE2EDuration="5.634498236s" totalImagesPullingTime="3.816325595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:06:17.632394136 +0000 UTC m=+2290.177343351" watchObservedRunningTime="2026-07-23 02:06:17.634498236 +0000 UTC m=+2290.179447462" Jul 23 02:06:23.628360 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:23.628329 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:06:29.764697 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:29.764652 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb"] Jul 23 02:06:29.768588 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:29.768561 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb" Jul 23 02:06:29.771228 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:29.771203 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l5l5h\"" Jul 23 02:06:29.774749 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:29.774721 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb"] Jul 23 02:06:29.883500 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:29.883465 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd7aa26-71d5-44be-af67-1bda588483a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb\" (UID: \"0dd7aa26-71d5-44be-af67-1bda588483a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb" Jul 23 02:06:29.983923 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:29.983891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd7aa26-71d5-44be-af67-1bda588483a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb\" (UID: \"0dd7aa26-71d5-44be-af67-1bda588483a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb" Jul 23 02:06:29.984510 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:29.984493 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd7aa26-71d5-44be-af67-1bda588483a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb\" (UID: \"0dd7aa26-71d5-44be-af67-1bda588483a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb" Jul 23 02:06:30.080973 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:30.080889 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb" Jul 23 02:06:30.223673 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:30.223645 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb"] Jul 23 02:06:30.225399 ip-10-0-131-117 kubenswrapper[2572]: W0723 02:06:30.225374 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd7aa26_71d5_44be_af67_1bda588483a6.slice/crio-f36545733a98951ab27b35be3874c056927166be2d9e857c6610cc8536ec4e5e WatchSource:0}: Error finding container f36545733a98951ab27b35be3874c056927166be2d9e857c6610cc8536ec4e5e: Status 404 returned error can't find the container with id f36545733a98951ab27b35be3874c056927166be2d9e857c6610cc8536ec4e5e Jul 23 02:06:30.691129 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:30.691087 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb" event={"ID":"0dd7aa26-71d5-44be-af67-1bda588483a6","Type":"ContainerStarted","Data":"0f97b85ffcf58d4db3d1f1c153f92c51bb250f40581c39d31c9d4d6f426f27c3"} Jul 23 02:06:30.691331 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:30.691141 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb" event={"ID":"0dd7aa26-71d5-44be-af67-1bda588483a6","Type":"ContainerStarted","Data":"f36545733a98951ab27b35be3874c056927166be2d9e857c6610cc8536ec4e5e"} Jul 23 02:06:30.706811 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:30.706719 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb" podStartSLOduration=1.706703161 podStartE2EDuration="1.706703161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:06:30.705440499 +0000 UTC m=+2303.250389674" watchObservedRunningTime="2026-07-23 02:06:30.706703161 +0000 UTC m=+2303.251652334" Jul 23 02:06:31.698237 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:31.698198 2572 generic.go:356] "Generic (PLEG): container finished" podID="0dd7aa26-71d5-44be-af67-1bda588483a6" containerID="0f97b85ffcf58d4db3d1f1c153f92c51bb250f40581c39d31c9d4d6f426f27c3" exitCode=0 Jul 23 02:06:31.698601 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:31.698262 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb" event={"ID":"0dd7aa26-71d5-44be-af67-1bda588483a6","Type":"ContainerDied","Data":"0f97b85ffcf58d4db3d1f1c153f92c51bb250f40581c39d31c9d4d6f426f27c3"} Jul 23 02:06:32.838710 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:32.838685 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb" Jul 23 02:06:33.009481 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:33.009400 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd7aa26-71d5-44be-af67-1bda588483a6-maas-api-service-ca\") pod \"0dd7aa26-71d5-44be-af67-1bda588483a6\" (UID: \"0dd7aa26-71d5-44be-af67-1bda588483a6\") " Jul 23 02:06:33.009795 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:33.009773 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd7aa26-71d5-44be-af67-1bda588483a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dd7aa26-71d5-44be-af67-1bda588483a6" (UID: "0dd7aa26-71d5-44be-af67-1bda588483a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 02:06:33.111009 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:33.110970 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd7aa26-71d5-44be-af67-1bda588483a6-maas-api-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:06:33.710128 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:33.710094 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb" Jul 23 02:06:33.710128 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:33.710111 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb" event={"ID":"0dd7aa26-71d5-44be-af67-1bda588483a6","Type":"ContainerDied","Data":"f36545733a98951ab27b35be3874c056927166be2d9e857c6610cc8536ec4e5e"} Jul 23 02:06:33.710346 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:33.710161 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f36545733a98951ab27b35be3874c056927166be2d9e857c6610cc8536ec4e5e" Jul 23 02:06:49.854674 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:49.854610 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb"] Jul 23 02:06:49.856672 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:49.856650 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dec09f-ff6cb"] Jul 23 02:06:49.975649 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:06:49.975605 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dd7aa26-71d5-44be-af67-1bda588483a6" path="/var/lib/kubelet/pods/0dd7aa26-71d5-44be-af67-1bda588483a6/volumes" Jul 23 02:07:08.254196 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:08.254103 2572 scope.go:117] "RemoveContainer" containerID="6fac75e91c00d42b2494963a2d08dc4236f93e14547a8eb7217ab38e7eac5c53" Jul 23 02:07:12.174130 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.174093 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8"] Jul 23 02:07:12.174627 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.174380 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" podUID="524cc1f0-9e2d-4b0f-a816-0172e326b49d" containerName="maas-api" containerID="cri-o://0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab" gracePeriod=30 Jul 23 02:07:12.437061 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.436979 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:07:12.565118 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.565079 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/524cc1f0-9e2d-4b0f-a816-0172e326b49d-maas-api-tls\") pod \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\" (UID: \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\") " Jul 23 02:07:12.565323 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.565147 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/524cc1f0-9e2d-4b0f-a816-0172e326b49d-openshift-service-ca\") pod \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\" (UID: \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\") " Jul 23 02:07:12.565323 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.565181 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjvzw\" (UniqueName: \"kubernetes.io/projected/524cc1f0-9e2d-4b0f-a816-0172e326b49d-kube-api-access-sjvzw\") pod \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\" (UID: \"524cc1f0-9e2d-4b0f-a816-0172e326b49d\") " Jul 23 02:07:12.566660 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.566595 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/524cc1f0-9e2d-4b0f-a816-0172e326b49d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "524cc1f0-9e2d-4b0f-a816-0172e326b49d" (UID: "524cc1f0-9e2d-4b0f-a816-0172e326b49d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 02:07:12.573226 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.573195 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/524cc1f0-9e2d-4b0f-a816-0172e326b49d-kube-api-access-sjvzw" (OuterVolumeSpecName: "kube-api-access-sjvzw") pod "524cc1f0-9e2d-4b0f-a816-0172e326b49d" (UID: "524cc1f0-9e2d-4b0f-a816-0172e326b49d"). InnerVolumeSpecName "kube-api-access-sjvzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 02:07:12.573867 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.573829 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/524cc1f0-9e2d-4b0f-a816-0172e326b49d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "524cc1f0-9e2d-4b0f-a816-0172e326b49d" (UID: "524cc1f0-9e2d-4b0f-a816-0172e326b49d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 02:07:12.666819 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.666782 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/524cc1f0-9e2d-4b0f-a816-0172e326b49d-maas-api-tls\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:07:12.666819 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.666815 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/524cc1f0-9e2d-4b0f-a816-0172e326b49d-openshift-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:07:12.666819 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.666824 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjvzw\" (UniqueName: \"kubernetes.io/projected/524cc1f0-9e2d-4b0f-a816-0172e326b49d-kube-api-access-sjvzw\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:07:12.923643 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.923608 2572 generic.go:356] "Generic (PLEG): container finished" podID="524cc1f0-9e2d-4b0f-a816-0172e326b49d" containerID="0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab" exitCode=0 Jul 23 02:07:12.923801 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.923694 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" Jul 23 02:07:12.923801 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.923689 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" event={"ID":"524cc1f0-9e2d-4b0f-a816-0172e326b49d","Type":"ContainerDied","Data":"0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab"} Jul 23 02:07:12.923950 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.923857 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8" event={"ID":"524cc1f0-9e2d-4b0f-a816-0172e326b49d","Type":"ContainerDied","Data":"3ad279830367693d9fab378b3b81515227692f048e8ba13317363bd80fc77832"} Jul 23 02:07:12.923950 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.923891 2572 scope.go:117] "RemoveContainer" containerID="0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab" Jul 23 02:07:12.933469 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.933452 2572 scope.go:117] "RemoveContainer" containerID="0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab" Jul 23 02:07:12.933762 ip-10-0-131-117 kubenswrapper[2572]: E0723 02:07:12.933743 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab\": container with ID starting with 0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab not found: ID does not exist" containerID="0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab" Jul 23 02:07:12.933835 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.933770 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab"} err="failed to get container status \"0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab\": rpc error: code = NotFound desc = could not find container \"0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab\": container with ID starting with 0d92cc51638edde711181a7211868659a836cb7fd39d8cb66ac05b2b8f10f1ab not found: ID does not exist" Jul 23 02:07:12.946936 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.946906 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8"] Jul 23 02:07:12.950772 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:12.950748 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf5ce0-66765f9589-8k7n8"] Jul 23 02:07:13.975231 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:13.975197 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="524cc1f0-9e2d-4b0f-a816-0172e326b49d" path="/var/lib/kubelet/pods/524cc1f0-9e2d-4b0f-a816-0172e326b49d/volumes" Jul 23 02:07:26.698604 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.698567 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9"] Jul 23 02:07:26.699109 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.698961 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="524cc1f0-9e2d-4b0f-a816-0172e326b49d" containerName="maas-api" Jul 23 02:07:26.699109 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.698973 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="524cc1f0-9e2d-4b0f-a816-0172e326b49d" containerName="maas-api" Jul 23 02:07:26.699109 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.698996 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd7aa26-71d5-44be-af67-1bda588483a6" containerName="revoke-keys" Jul 23 02:07:26.699109 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.699001 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd7aa26-71d5-44be-af67-1bda588483a6" containerName="revoke-keys" Jul 23 02:07:26.699109 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.699052 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="524cc1f0-9e2d-4b0f-a816-0172e326b49d" containerName="maas-api" Jul 23 02:07:26.699109 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.699062 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd7aa26-71d5-44be-af67-1bda588483a6" containerName="revoke-keys" Jul 23 02:07:26.702572 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.702554 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:26.705678 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.705654 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rzlfw\"" Jul 23 02:07:26.705884 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.705871 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-1dd296-serving-cert\"" Jul 23 02:07:26.715599 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.715571 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9"] Jul 23 02:07:26.790699 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.790655 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974k5\" (UniqueName: \"kubernetes.io/projected/15dad82f-50db-43f6-96f3-3a65ed46fbd0-kube-api-access-974k5\") pod \"maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:26.790892 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.790729 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15dad82f-50db-43f6-96f3-3a65ed46fbd0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:26.790892 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.790772 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dad82f-50db-43f6-96f3-3a65ed46fbd0-maas-api-tls\") pod \"maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:26.891340 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.891305 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dad82f-50db-43f6-96f3-3a65ed46fbd0-maas-api-tls\") pod \"maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:26.891683 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.891368 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974k5\" (UniqueName: \"kubernetes.io/projected/15dad82f-50db-43f6-96f3-3a65ed46fbd0-kube-api-access-974k5\") pod \"maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:26.891683 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.891414 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15dad82f-50db-43f6-96f3-3a65ed46fbd0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:26.891683 ip-10-0-131-117 kubenswrapper[2572]: E0723 02:07:26.891483 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-serving-cert: secret "maas-api-e2e-shared-a-1dd296-serving-cert" not found Jul 23 02:07:26.891683 ip-10-0-131-117 kubenswrapper[2572]: E0723 02:07:26.891578 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15dad82f-50db-43f6-96f3-3a65ed46fbd0-maas-api-tls podName:15dad82f-50db-43f6-96f3-3a65ed46fbd0 nodeName:}" failed. No retries permitted until 2026-07-23 02:07:27.391554711 +0000 UTC m=+2359.936503881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15dad82f-50db-43f6-96f3-3a65ed46fbd0-maas-api-tls") pod "maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" (UID: "15dad82f-50db-43f6-96f3-3a65ed46fbd0") : secret "maas-api-e2e-shared-a-1dd296-serving-cert" not found Jul 23 02:07:26.892460 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.892438 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15dad82f-50db-43f6-96f3-3a65ed46fbd0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:26.900933 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:26.900904 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-974k5\" (UniqueName: \"kubernetes.io/projected/15dad82f-50db-43f6-96f3-3a65ed46fbd0-kube-api-access-974k5\") pod \"maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:27.395680 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:27.395622 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dad82f-50db-43f6-96f3-3a65ed46fbd0-maas-api-tls\") pod \"maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:27.398203 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:27.398175 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dad82f-50db-43f6-96f3-3a65ed46fbd0-maas-api-tls\") pod \"maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:27.612292 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:27.612251 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:27.754859 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:27.754834 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9"] Jul 23 02:07:27.756665 ip-10-0-131-117 kubenswrapper[2572]: W0723 02:07:27.756616 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15dad82f_50db_43f6_96f3_3a65ed46fbd0.slice/crio-8e107de65f2be6ab09cdd55a7d7bd90de974b48a5dc029a84b699f74f778d429 WatchSource:0}: Error finding container 8e107de65f2be6ab09cdd55a7d7bd90de974b48a5dc029a84b699f74f778d429: Status 404 returned error can't find the container with id 8e107de65f2be6ab09cdd55a7d7bd90de974b48a5dc029a84b699f74f778d429 Jul 23 02:07:28.008300 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:28.008217 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" event={"ID":"15dad82f-50db-43f6-96f3-3a65ed46fbd0","Type":"ContainerStarted","Data":"8e107de65f2be6ab09cdd55a7d7bd90de974b48a5dc029a84b699f74f778d429"} Jul 23 02:07:29.015205 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:29.015162 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" event={"ID":"15dad82f-50db-43f6-96f3-3a65ed46fbd0","Type":"ContainerStarted","Data":"eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c"} Jul 23 02:07:29.015585 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:29.015324 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:29.032343 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:29.032283 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" podStartSLOduration=2.554109729 podStartE2EDuration="3.03226884s" totalImagesPullingTime="478.159111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:07:29.030492657 +0000 UTC m=+2361.575441834" watchObservedRunningTime="2026-07-23 02:07:29.03226884 +0000 UTC m=+2361.577218012" Jul 23 02:07:34.766883 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.766847 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48"] Jul 23 02:07:34.771146 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.771121 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:07:34.774784 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.774761 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-5fbcb9-serving-cert\"" Jul 23 02:07:34.781275 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.781248 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48"] Jul 23 02:07:34.861921 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.861876 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d352da3-bb89-4735-883d-eb2aa33f5fe3-maas-api-tls\") pod \"maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48\" (UID: \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:07:34.862107 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.861960 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjw2l\" (UniqueName: \"kubernetes.io/projected/9d352da3-bb89-4735-883d-eb2aa33f5fe3-kube-api-access-fjw2l\") pod \"maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48\" (UID: \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:07:34.862107 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.861989 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d352da3-bb89-4735-883d-eb2aa33f5fe3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48\" (UID: \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:07:34.963245 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.963208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjw2l\" (UniqueName: \"kubernetes.io/projected/9d352da3-bb89-4735-883d-eb2aa33f5fe3-kube-api-access-fjw2l\") pod \"maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48\" (UID: \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:07:34.963245 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.963255 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d352da3-bb89-4735-883d-eb2aa33f5fe3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48\" (UID: \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:07:34.963529 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.963303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d352da3-bb89-4735-883d-eb2aa33f5fe3-maas-api-tls\") pod \"maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48\" (UID: \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:07:34.964050 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.964025 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d352da3-bb89-4735-883d-eb2aa33f5fe3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48\" (UID: \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:07:34.965706 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.965688 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d352da3-bb89-4735-883d-eb2aa33f5fe3-maas-api-tls\") pod \"maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48\" (UID: \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:07:34.974784 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:34.974750 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjw2l\" (UniqueName: \"kubernetes.io/projected/9d352da3-bb89-4735-883d-eb2aa33f5fe3-kube-api-access-fjw2l\") pod \"maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48\" (UID: \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:07:35.027742 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:35.027659 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:07:35.083783 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:35.083752 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:07:35.228812 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:35.228776 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48"] Jul 23 02:07:35.230014 ip-10-0-131-117 kubenswrapper[2572]: W0723 02:07:35.229985 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d352da3_bb89_4735_883d_eb2aa33f5fe3.slice/crio-e395b16cff5739e5e7ca0f41832e71bdf3ed6c6689902391dbd0e181fce366b6 WatchSource:0}: Error finding container e395b16cff5739e5e7ca0f41832e71bdf3ed6c6689902391dbd0e181fce366b6: Status 404 returned error can't find the container with id e395b16cff5739e5e7ca0f41832e71bdf3ed6c6689902391dbd0e181fce366b6 Jul 23 02:07:36.062209 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:36.062107 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" event={"ID":"9d352da3-bb89-4735-883d-eb2aa33f5fe3","Type":"ContainerStarted","Data":"6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b"} Jul 23 02:07:36.062209 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:36.062196 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" event={"ID":"9d352da3-bb89-4735-883d-eb2aa33f5fe3","Type":"ContainerStarted","Data":"e395b16cff5739e5e7ca0f41832e71bdf3ed6c6689902391dbd0e181fce366b6"} Jul 23 02:07:36.062770 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:36.062226 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:07:36.079534 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:36.079476 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" podStartSLOduration=1.6991018169999998 podStartE2EDuration="2.079461325s" totalImagesPullingTime="380.359508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 02:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:07:36.077659915 +0000 UTC m=+2368.622609078" watchObservedRunningTime="2026-07-23 02:07:36.079461325 +0000 UTC m=+2368.624410497" Jul 23 02:07:42.073973 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:07:42.073942 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:08:08.276131 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:08:08.276100 2572 scope.go:117] "RemoveContainer" containerID="9771b38326858b907aa50bd1c5ad5a6010c2ba3ac09dd0abda760cb11819a6b3" Jul 23 02:12:08.326314 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:12:08.326280 2572 scope.go:117] "RemoveContainer" containerID="e6eebd5d1831d55e6353cf8612d84df5b2da16e113085d80bef52d17c1337c2d" Jul 23 02:12:08.338535 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:12:08.338506 2572 scope.go:117] "RemoveContainer" containerID="a360378fc3e94dce10dffa01616f394083e9951e73eb39c4009c39611e1ea192" Jul 23 02:13:08.361605 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:08.361521 2572 scope.go:117] "RemoveContainer" containerID="0f97b85ffcf58d4db3d1f1c153f92c51bb250f40581c39d31c9d4d6f426f27c3" Jul 23 02:13:36.199874 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:36.199833 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9"] Jul 23 02:13:36.200397 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:36.200125 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" podUID="15dad82f-50db-43f6-96f3-3a65ed46fbd0" containerName="maas-api" containerID="cri-o://eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c" gracePeriod=30 Jul 23 02:13:36.469856 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:36.469829 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:13:36.562164 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:36.562129 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dad82f-50db-43f6-96f3-3a65ed46fbd0-maas-api-tls\") pod \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " Jul 23 02:13:36.562164 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:36.562169 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-974k5\" (UniqueName: \"kubernetes.io/projected/15dad82f-50db-43f6-96f3-3a65ed46fbd0-kube-api-access-974k5\") pod \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " Jul 23 02:13:36.562370 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:36.562199 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15dad82f-50db-43f6-96f3-3a65ed46fbd0-openshift-service-ca\") pod \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\" (UID: \"15dad82f-50db-43f6-96f3-3a65ed46fbd0\") " Jul 23 02:13:36.562662 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:36.562603 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15dad82f-50db-43f6-96f3-3a65ed46fbd0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15dad82f-50db-43f6-96f3-3a65ed46fbd0" (UID: "15dad82f-50db-43f6-96f3-3a65ed46fbd0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 02:13:36.564304 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:36.564275 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15dad82f-50db-43f6-96f3-3a65ed46fbd0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15dad82f-50db-43f6-96f3-3a65ed46fbd0" (UID: "15dad82f-50db-43f6-96f3-3a65ed46fbd0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 02:13:36.564423 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:36.564279 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15dad82f-50db-43f6-96f3-3a65ed46fbd0-kube-api-access-974k5" (OuterVolumeSpecName: "kube-api-access-974k5") pod "15dad82f-50db-43f6-96f3-3a65ed46fbd0" (UID: "15dad82f-50db-43f6-96f3-3a65ed46fbd0"). InnerVolumeSpecName "kube-api-access-974k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 02:13:36.663178 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:36.663148 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dad82f-50db-43f6-96f3-3a65ed46fbd0-maas-api-tls\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:13:36.663178 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:36.663176 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974k5\" (UniqueName: \"kubernetes.io/projected/15dad82f-50db-43f6-96f3-3a65ed46fbd0-kube-api-access-974k5\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:13:36.663178 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:36.663185 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15dad82f-50db-43f6-96f3-3a65ed46fbd0-openshift-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:13:37.008286 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:37.008253 2572 generic.go:356] "Generic (PLEG): container finished" podID="15dad82f-50db-43f6-96f3-3a65ed46fbd0" containerID="eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c" exitCode=0 Jul 23 02:13:37.008477 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:37.008295 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" Jul 23 02:13:37.008477 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:37.008332 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" event={"ID":"15dad82f-50db-43f6-96f3-3a65ed46fbd0","Type":"ContainerDied","Data":"eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c"} Jul 23 02:13:37.008477 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:37.008383 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9" event={"ID":"15dad82f-50db-43f6-96f3-3a65ed46fbd0","Type":"ContainerDied","Data":"8e107de65f2be6ab09cdd55a7d7bd90de974b48a5dc029a84b699f74f778d429"} Jul 23 02:13:37.008477 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:37.008408 2572 scope.go:117] "RemoveContainer" containerID="eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c" Jul 23 02:13:37.018135 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:37.018118 2572 scope.go:117] "RemoveContainer" containerID="eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c" Jul 23 02:13:37.018417 ip-10-0-131-117 kubenswrapper[2572]: E0723 02:13:37.018399 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c\": container with ID starting with eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c not found: ID does not exist" containerID="eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c" Jul 23 02:13:37.018460 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:37.018426 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c"} err="failed to get container status \"eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c\": rpc error: code = NotFound desc = could not find container \"eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c\": container with ID starting with eb7a973bda5a87f7ce2b220b6a99ca93552ee74fa9c1144b8a815ba93994031c not found: ID does not exist" Jul 23 02:13:37.030325 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:37.030290 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9"] Jul 23 02:13:37.035516 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:37.035485 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1dd296-7c59c57dc-wqgt9"] Jul 23 02:13:37.975994 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:37.975958 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15dad82f-50db-43f6-96f3-3a65ed46fbd0" path="/var/lib/kubelet/pods/15dad82f-50db-43f6-96f3-3a65ed46fbd0/volumes" Jul 23 02:13:57.808153 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:57.808116 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48"] Jul 23 02:13:57.808667 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:57.808364 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" podUID="9d352da3-bb89-4735-883d-eb2aa33f5fe3" containerName="maas-api" containerID="cri-o://6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b" gracePeriod=30 Jul 23 02:13:58.063037 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.062970 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:13:58.127404 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.127371 2572 generic.go:356] "Generic (PLEG): container finished" podID="9d352da3-bb89-4735-883d-eb2aa33f5fe3" containerID="6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b" exitCode=0 Jul 23 02:13:58.127586 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.127421 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" Jul 23 02:13:58.127586 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.127439 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" event={"ID":"9d352da3-bb89-4735-883d-eb2aa33f5fe3","Type":"ContainerDied","Data":"6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b"} Jul 23 02:13:58.127586 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.127483 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48" event={"ID":"9d352da3-bb89-4735-883d-eb2aa33f5fe3","Type":"ContainerDied","Data":"e395b16cff5739e5e7ca0f41832e71bdf3ed6c6689902391dbd0e181fce366b6"} Jul 23 02:13:58.127586 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.127505 2572 scope.go:117] "RemoveContainer" containerID="6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b" Jul 23 02:13:58.136694 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.136673 2572 scope.go:117] "RemoveContainer" containerID="6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b" Jul 23 02:13:58.137034 ip-10-0-131-117 kubenswrapper[2572]: E0723 02:13:58.137011 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b\": container with ID starting with 6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b not found: ID does not exist" containerID="6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b" Jul 23 02:13:58.137146 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.137040 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b"} err="failed to get container status \"6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b\": rpc error: code = NotFound desc = could not find container \"6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b\": container with ID starting with 6b85899663399841db3606ecd25bfdda81d5b99821d0d41126fcd648e6a3b34b not found: ID does not exist" Jul 23 02:13:58.148401 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.148375 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjw2l\" (UniqueName: \"kubernetes.io/projected/9d352da3-bb89-4735-883d-eb2aa33f5fe3-kube-api-access-fjw2l\") pod \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\" (UID: \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\") " Jul 23 02:13:58.148518 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.148423 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d352da3-bb89-4735-883d-eb2aa33f5fe3-maas-api-tls\") pod \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\" (UID: \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\") " Jul 23 02:13:58.148518 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.148442 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d352da3-bb89-4735-883d-eb2aa33f5fe3-openshift-service-ca\") pod \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\" (UID: \"9d352da3-bb89-4735-883d-eb2aa33f5fe3\") " Jul 23 02:13:58.148877 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.148850 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d352da3-bb89-4735-883d-eb2aa33f5fe3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d352da3-bb89-4735-883d-eb2aa33f5fe3" (UID: "9d352da3-bb89-4735-883d-eb2aa33f5fe3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 02:13:58.150467 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.150441 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d352da3-bb89-4735-883d-eb2aa33f5fe3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d352da3-bb89-4735-883d-eb2aa33f5fe3" (UID: "9d352da3-bb89-4735-883d-eb2aa33f5fe3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 02:13:58.150552 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.150467 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d352da3-bb89-4735-883d-eb2aa33f5fe3-kube-api-access-fjw2l" (OuterVolumeSpecName: "kube-api-access-fjw2l") pod "9d352da3-bb89-4735-883d-eb2aa33f5fe3" (UID: "9d352da3-bb89-4735-883d-eb2aa33f5fe3"). InnerVolumeSpecName "kube-api-access-fjw2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 02:13:58.249270 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.249231 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjw2l\" (UniqueName: \"kubernetes.io/projected/9d352da3-bb89-4735-883d-eb2aa33f5fe3-kube-api-access-fjw2l\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:13:58.249270 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.249265 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d352da3-bb89-4735-883d-eb2aa33f5fe3-maas-api-tls\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:13:58.249270 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.249275 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d352da3-bb89-4735-883d-eb2aa33f5fe3-openshift-service-ca\") on node \"ip-10-0-131-117.ec2.internal\" DevicePath \"\"" Jul 23 02:13:58.451267 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.451230 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48"] Jul 23 02:13:58.462410 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:58.457378 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5fbcb9-7d846c6ff7-dlv48"] Jul 23 02:13:59.975933 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:13:59.975901 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d352da3-bb89-4735-883d-eb2aa33f5fe3" path="/var/lib/kubelet/pods/9d352da3-bb89-4735-883d-eb2aa33f5fe3/volumes" Jul 23 02:14:15.416716 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:15.416686 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cfkpl_787d4d5b-c973-4a3c-956b-c53163ef2b3a/manager/0.log" Jul 23 02:14:15.546347 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:15.546315 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-258ws_8c96e9f7-295f-401d-8459-b336023730c3/manager/0.log" Jul 23 02:14:15.693523 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:15.693440 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-cc8bb7484-qss2q_3f761efa-e6f1-4dbe-8702-0c2a1bab3730/manager/0.log" Jul 23 02:14:15.812128 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:15.812104 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tf2sm_d5265802-05d7-42fa-865f-70d0f15023ee/server/0.log" Jul 23 02:14:15.936949 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:15.936914 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xkv9t_22f21c1f-dbd9-4a70-9606-0217f81be9fb/manager/0.log" Jul 23 02:14:16.060295 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:16.060214 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr_7ea2a247-3878-47bf-b76c-e0749a73a665/manager/0.log" Jul 23 02:14:16.067593 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:16.067569 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr_7ea2a247-3878-47bf-b76c-e0749a73a665/copy-manifests/0.log" Jul 23 02:14:16.839775 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:16.839745 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746185-8fvcc_6d2cafea-5819-45e3-b183-fba8ac187c93/cleanup/0.log" Jul 23 02:14:17.077956 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:17.077924 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-s6q4v_20ead147-5678-4a18-bc75-c4ecf689b5b4/postgres/0.log" Jul 23 02:14:17.911366 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:17.911338 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm_806a17ab-62e4-4ce5-b109-f51c7f50f4f9/pull/0.log" Jul 23 02:14:17.918581 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:17.918556 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm_806a17ab-62e4-4ce5-b109-f51c7f50f4f9/extract/0.log" Jul 23 02:14:17.924867 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:17.924844 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm_806a17ab-62e4-4ce5-b109-f51c7f50f4f9/util/0.log" Jul 23 02:14:18.052185 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:18.052152 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp_bea38df4-c56c-44f3-959f-9d948eeb67aa/pull/0.log" Jul 23 02:14:18.058045 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:18.058027 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp_bea38df4-c56c-44f3-959f-9d948eeb67aa/extract/0.log" Jul 23 02:14:18.064190 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:18.064168 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp_bea38df4-c56c-44f3-959f-9d948eeb67aa/util/0.log" Jul 23 02:14:18.189987 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:18.189900 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99_e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3/util/0.log" Jul 23 02:14:18.196766 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:18.196742 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99_e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3/pull/0.log" Jul 23 02:14:18.203366 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:18.203344 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99_e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3/extract/0.log" Jul 23 02:14:18.320878 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:18.320848 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf_adfb6570-f16f-4985-b6db-2a07c0aab0e9/util/0.log" Jul 23 02:14:18.326415 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:18.326394 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf_adfb6570-f16f-4985-b6db-2a07c0aab0e9/pull/0.log" Jul 23 02:14:18.331902 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:18.331879 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf_adfb6570-f16f-4985-b6db-2a07c0aab0e9/extract/0.log" Jul 23 02:14:19.344185 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:19.344157 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-zrff7_860a10d0-dde0-48a4-b4fa-9923f6b15e46/manager/0.log" Jul 23 02:14:19.842559 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:19.842529 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gnjxd_6adbbd58-4e62-4244-9840-970e0b9d740a/discovery/0.log" Jul 23 02:14:20.084113 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:20.084082 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85497ff65b-wnv44_aa85ab52-95a8-4d0f-88e3-a9d4d261524f/kube-auth-proxy/0.log" Jul 23 02:14:20.450369 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:20.450338 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-4whc6_fe40b256-bab4-48de-812f-f22624468832/payload-processing/0.log" Jul 23 02:14:20.945752 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:20.945723 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-f78c8_290af6d9-abb7-4b07-9935-c4e4338a6c3f/main/0.log" Jul 23 02:14:21.057747 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:21.057714 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-5csh4_9027281c-43dc-4686-b027-dd88cce64e08/main/0.log" Jul 23 02:14:30.939295 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:30.939206 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vvpsg_f94eece0-4626-4e4e-8c9f-d8aab764fe95/global-pull-secret-syncer/0.log" Jul 23 02:14:31.068331 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:31.068280 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zvjp5_53c3ed25-3b0b-46b0-a5ba-1b0b87900743/konnectivity-agent/0.log" Jul 23 02:14:31.111006 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:31.110980 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-117.ec2.internal_09f69a889a832c91f7f75eb1f3452d24/haproxy/0.log" Jul 23 02:14:35.629100 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:35.629067 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm_806a17ab-62e4-4ce5-b109-f51c7f50f4f9/extract/0.log" Jul 23 02:14:35.651888 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:35.651859 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm_806a17ab-62e4-4ce5-b109-f51c7f50f4f9/util/0.log" Jul 23 02:14:35.675594 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:35.675565 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592kqwm_806a17ab-62e4-4ce5-b109-f51c7f50f4f9/pull/0.log" Jul 23 02:14:35.713859 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:35.713819 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp_bea38df4-c56c-44f3-959f-9d948eeb67aa/extract/0.log" Jul 23 02:14:35.736569 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:35.736542 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp_bea38df4-c56c-44f3-959f-9d948eeb67aa/util/0.log" Jul 23 02:14:35.759018 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:35.758994 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07sngp_bea38df4-c56c-44f3-959f-9d948eeb67aa/pull/0.log" Jul 23 02:14:35.787527 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:35.787496 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99_e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3/extract/0.log" Jul 23 02:14:35.812089 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:35.812058 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99_e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3/util/0.log" Jul 23 02:14:35.833619 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:35.833586 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpn99_e1ee4acd-7842-45dc-8dcf-0a266c5e1fd3/pull/0.log" Jul 23 02:14:35.862657 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:35.862608 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf_adfb6570-f16f-4985-b6db-2a07c0aab0e9/extract/0.log" Jul 23 02:14:35.884212 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:35.884124 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf_adfb6570-f16f-4985-b6db-2a07c0aab0e9/util/0.log" Jul 23 02:14:35.906744 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:35.906719 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mpdxf_adfb6570-f16f-4985-b6db-2a07c0aab0e9/pull/0.log" Jul 23 02:14:36.802817 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:36.802777 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-zrff7_860a10d0-dde0-48a4-b4fa-9923f6b15e46/manager/0.log" Jul 23 02:14:38.787116 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:38.787087 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kjxjh_7cad8448-e3b6-4e82-bb12-520a9e2adeec/node-exporter/0.log" Jul 23 02:14:38.806643 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:38.806604 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kjxjh_7cad8448-e3b6-4e82-bb12-520a9e2adeec/kube-rbac-proxy/0.log" Jul 23 02:14:38.827959 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:38.827935 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kjxjh_7cad8448-e3b6-4e82-bb12-520a9e2adeec/init-textfile/0.log" Jul 23 02:14:39.271736 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:39.271706 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-6rxf4_55f9819e-440c-45f9-bcd9-72ef8ed5e14b/prometheus-operator-admission-webhook/0.log" Jul 23 02:14:40.640850 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.640817 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-xmkj5_b4514d23-1e8e-4fc5-9f49-ab61bb5c929a/networking-console-plugin/0.log" Jul 23 02:14:40.657591 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.657557 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2"] Jul 23 02:14:40.658044 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.658026 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15dad82f-50db-43f6-96f3-3a65ed46fbd0" containerName="maas-api" Jul 23 02:14:40.658044 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.658046 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="15dad82f-50db-43f6-96f3-3a65ed46fbd0" containerName="maas-api" Jul 23 02:14:40.658044 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.658055 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d352da3-bb89-4735-883d-eb2aa33f5fe3" containerName="maas-api" Jul 23 02:14:40.658199 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.658061 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d352da3-bb89-4735-883d-eb2aa33f5fe3" containerName="maas-api" Jul 23 02:14:40.658199 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.658135 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="15dad82f-50db-43f6-96f3-3a65ed46fbd0" containerName="maas-api" Jul 23 02:14:40.658199 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.658145 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d352da3-bb89-4735-883d-eb2aa33f5fe3" containerName="maas-api" Jul 23 02:14:40.661483 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.661467 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.663779 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.663755 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-54hzb\"/\"openshift-service-ca.crt\"" Jul 23 02:14:40.663956 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.663936 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-54hzb\"/\"default-dockercfg-nz4lg\"" Jul 23 02:14:40.664693 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.664679 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-54hzb\"/\"kube-root-ca.crt\"" Jul 23 02:14:40.670064 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.670042 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2"] Jul 23 02:14:40.710630 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.710581 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05b3ecfc-b817-4738-a84a-d7039334046e-sys\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.710862 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.710669 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05b3ecfc-b817-4738-a84a-d7039334046e-lib-modules\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.710862 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.710753 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/05b3ecfc-b817-4738-a84a-d7039334046e-podres\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.710862 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.710810 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/05b3ecfc-b817-4738-a84a-d7039334046e-proc\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.711012 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.710856 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw9jq\" (UniqueName: \"kubernetes.io/projected/05b3ecfc-b817-4738-a84a-d7039334046e-kube-api-access-pw9jq\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.811381 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.811331 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/05b3ecfc-b817-4738-a84a-d7039334046e-podres\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.811381 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.811397 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/05b3ecfc-b817-4738-a84a-d7039334046e-proc\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.811698 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.811415 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw9jq\" (UniqueName: \"kubernetes.io/projected/05b3ecfc-b817-4738-a84a-d7039334046e-kube-api-access-pw9jq\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.811698 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.811433 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05b3ecfc-b817-4738-a84a-d7039334046e-sys\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.811698 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.811468 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05b3ecfc-b817-4738-a84a-d7039334046e-lib-modules\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.811698 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.811501 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/05b3ecfc-b817-4738-a84a-d7039334046e-podres\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.811698 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.811544 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05b3ecfc-b817-4738-a84a-d7039334046e-sys\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.811698 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.811569 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05b3ecfc-b817-4738-a84a-d7039334046e-lib-modules\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.811698 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.811551 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/05b3ecfc-b817-4738-a84a-d7039334046e-proc\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.820078 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.820049 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw9jq\" (UniqueName: \"kubernetes.io/projected/05b3ecfc-b817-4738-a84a-d7039334046e-kube-api-access-pw9jq\") pod \"perf-node-gather-daemonset-trmb2\" (UID: \"05b3ecfc-b817-4738-a84a-d7039334046e\") " pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:40.973117 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:40.973022 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:41.111993 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:41.111969 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2"] Jul 23 02:14:41.113904 ip-10-0-131-117 kubenswrapper[2572]: W0723 02:14:41.113876 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod05b3ecfc_b817_4738_a84a_d7039334046e.slice/crio-adce62dfd2cd36155ca9163b2c13e3d13b2e7c752e2d9557fa45af4f5067829d WatchSource:0}: Error finding container adce62dfd2cd36155ca9163b2c13e3d13b2e7c752e2d9557fa45af4f5067829d: Status 404 returned error can't find the container with id adce62dfd2cd36155ca9163b2c13e3d13b2e7c752e2d9557fa45af4f5067829d Jul 23 02:14:41.354778 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:41.354736 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" event={"ID":"05b3ecfc-b817-4738-a84a-d7039334046e","Type":"ContainerStarted","Data":"9ab1fc6d246a096d0b6d24426b1862d1ebeb6d4db60b8fc704c931f0ae7f4f83"} Jul 23 02:14:41.354978 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:41.354794 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" event={"ID":"05b3ecfc-b817-4738-a84a-d7039334046e","Type":"ContainerStarted","Data":"adce62dfd2cd36155ca9163b2c13e3d13b2e7c752e2d9557fa45af4f5067829d"} Jul 23 02:14:41.354978 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:41.354816 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:41.370307 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:41.370260 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" podStartSLOduration=1.370243096 podStartE2EDuration="1.370243096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 02:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 02:14:41.368994797 +0000 UTC m=+2793.913943995" watchObservedRunningTime="2026-07-23 02:14:41.370243096 +0000 UTC m=+2793.915192269" Jul 23 02:14:41.807718 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:41.807690 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7bfc6b5b57-5mlpw_ecf4c727-68a7-473e-a8a0-83c04ec27eb0/console/0.log" Jul 23 02:14:41.838582 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:41.838555 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-rcvbl_eaabba99-6e4f-4c8e-9ff3-f49edbafcc2a/download-server/0.log" Jul 23 02:14:43.579798 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:43.579769 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fdh5s_75178789-2a57-449a-8fed-79136d817399/dns/0.log" Jul 23 02:14:43.603532 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:43.603502 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fdh5s_75178789-2a57-449a-8fed-79136d817399/kube-rbac-proxy/0.log" Jul 23 02:14:43.677415 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:43.677379 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ssx5d_820cce87-a018-4751-a835-a045e7ccfd8e/dns-node-resolver/0.log" Jul 23 02:14:44.214855 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:44.214818 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59bf8bfc75-qgnrr_a444081d-b059-444a-831f-bac40ce1f771/registry/0.log" Jul 23 02:14:44.286177 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:44.286149 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rz68t_50570df0-9213-4039-ab5c-a1347dc88854/node-ca/0.log" Jul 23 02:14:45.310721 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:45.310689 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gnjxd_6adbbd58-4e62-4244-9840-970e0b9d740a/discovery/0.log" Jul 23 02:14:45.357501 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:45.357472 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85497ff65b-wnv44_aa85ab52-95a8-4d0f-88e3-a9d4d261524f/kube-auth-proxy/0.log" Jul 23 02:14:45.485657 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:45.485613 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-4whc6_fe40b256-bab4-48de-812f-f22624468832/payload-processing/0.log" Jul 23 02:14:46.192694 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:46.192663 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ghjqn_3bef66d3-4454-4b73-b6ce-a9f8b034622f/serve-healthcheck-canary/0.log" Jul 23 02:14:46.711099 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:46.711067 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2kg5_21fe7634-3a93-41eb-8801-6ce83ee96fdc/kube-rbac-proxy/0.log" Jul 23 02:14:46.731093 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:46.731065 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2kg5_21fe7634-3a93-41eb-8801-6ce83ee96fdc/exporter/0.log" Jul 23 02:14:46.751729 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:46.751705 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2kg5_21fe7634-3a93-41eb-8801-6ce83ee96fdc/extractor/0.log" Jul 23 02:14:47.369587 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:47.369560 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-54hzb/perf-node-gather-daemonset-trmb2" Jul 23 02:14:48.991575 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:48.991544 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cfkpl_787d4d5b-c973-4a3c-956b-c53163ef2b3a/manager/0.log" Jul 23 02:14:49.183032 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:49.182997 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-258ws_8c96e9f7-295f-401d-8459-b336023730c3/manager/0.log" Jul 23 02:14:49.290293 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:49.290192 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-cc8bb7484-qss2q_3f761efa-e6f1-4dbe-8702-0c2a1bab3730/manager/0.log" Jul 23 02:14:49.311287 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:49.311256 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tf2sm_d5265802-05d7-42fa-865f-70d0f15023ee/server/0.log" Jul 23 02:14:49.386066 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:49.386018 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xkv9t_22f21c1f-dbd9-4a70-9606-0217f81be9fb/manager/0.log" Jul 23 02:14:49.463576 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:49.463534 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr_7ea2a247-3878-47bf-b76c-e0749a73a665/manager/0.log" Jul 23 02:14:49.482614 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:49.482583 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bf9cbfb7c-9fqbr_7ea2a247-3878-47bf-b76c-e0749a73a665/copy-manifests/0.log" Jul 23 02:14:51.652827 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:51.652796 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gnjxd_6adbbd58-4e62-4244-9840-970e0b9d740a/discovery/0.log" Jul 23 02:14:51.684744 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:51.684713 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85497ff65b-wnv44_aa85ab52-95a8-4d0f-88e3-a9d4d261524f/kube-auth-proxy/0.log" Jul 23 02:14:51.804178 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:51.804132 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-4whc6_fe40b256-bab4-48de-812f-f22624468832/payload-processing/0.log" Jul 23 02:14:53.059894 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:53.059808 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gnjxd_6adbbd58-4e62-4244-9840-970e0b9d740a/discovery/0.log" Jul 23 02:14:53.093374 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:53.093342 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85497ff65b-wnv44_aa85ab52-95a8-4d0f-88e3-a9d4d261524f/kube-auth-proxy/0.log" Jul 23 02:14:53.208160 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:53.208128 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-4whc6_fe40b256-bab4-48de-812f-f22624468832/payload-processing/0.log" Jul 23 02:14:56.250099 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:56.250069 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-tnpqp_1119539a-0bf1-44ac-98b2-1e53fe488cd5/migrator/0.log" Jul 23 02:14:56.272295 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:56.272265 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-tnpqp_1119539a-0bf1-44ac-98b2-1e53fe488cd5/graceful-termination/0.log" Jul 23 02:14:57.952012 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:57.951979 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xfprg_edab0343-4377-4ac0-8106-b39733beefc7/kube-multus-additional-cni-plugins/0.log" Jul 23 02:14:57.979346 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:57.979318 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xfprg_edab0343-4377-4ac0-8106-b39733beefc7/egress-router-binary-copy/0.log" Jul 23 02:14:57.997310 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:57.997286 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xfprg_edab0343-4377-4ac0-8106-b39733beefc7/cni-plugins/0.log" Jul 23 02:14:58.019040 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:58.019010 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xfprg_edab0343-4377-4ac0-8106-b39733beefc7/bond-cni-plugin/0.log" Jul 23 02:14:58.041024 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:58.040998 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xfprg_edab0343-4377-4ac0-8106-b39733beefc7/routeoverride-cni/0.log" Jul 23 02:14:58.064234 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:58.064208 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xfprg_edab0343-4377-4ac0-8106-b39733beefc7/whereabouts-cni-bincopy/0.log" Jul 23 02:14:58.087021 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:58.086995 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xfprg_edab0343-4377-4ac0-8106-b39733beefc7/whereabouts-cni/0.log" Jul 23 02:14:58.123364 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:58.123334 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-j5hcq_2a59326e-1357-42d7-96ae-4d6e43dabf24/kube-multus/0.log" Jul 23 02:14:58.298876 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:58.298797 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sgszg_62b44b95-e27a-42a9-b058-0e0363718750/network-metrics-daemon/0.log" Jul 23 02:14:58.316024 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:58.316001 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sgszg_62b44b95-e27a-42a9-b058-0e0363718750/kube-rbac-proxy/0.log" Jul 23 02:14:59.124475 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:59.124442 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6v4s_2fa70596-1837-4f06-b94c-88558d60b65e/ovn-controller/0.log" Jul 23 02:14:59.141307 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:59.141280 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6v4s_2fa70596-1837-4f06-b94c-88558d60b65e/ovn-acl-logging/0.log" Jul 23 02:14:59.164021 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:59.163991 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6v4s_2fa70596-1837-4f06-b94c-88558d60b65e/kube-rbac-proxy-node/0.log" Jul 23 02:14:59.185664 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:59.185622 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6v4s_2fa70596-1837-4f06-b94c-88558d60b65e/kube-rbac-proxy-ovn-metrics/0.log" Jul 23 02:14:59.205553 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:59.205525 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6v4s_2fa70596-1837-4f06-b94c-88558d60b65e/northd/0.log" Jul 23 02:14:59.228077 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:59.228053 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6v4s_2fa70596-1837-4f06-b94c-88558d60b65e/nbdb/0.log" Jul 23 02:14:59.249058 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:59.249032 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6v4s_2fa70596-1837-4f06-b94c-88558d60b65e/sbdb/0.log" Jul 23 02:14:59.304532 ip-10-0-131-117 kubenswrapper[2572]: I0723 02:14:59.304505 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6v4s_2fa70596-1837-4f06-b94c-88558d60b65e/ovnkube-controller/0.log"