Sep 05 19:42:59.392149 ip-10-0-143-252 systemd[1]: Starting Kubernetes Kubelet... Sep 05 19:42:59.823668 ip-10-0-143-252 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 05 19:42:59.823668 ip-10-0-143-252 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 05 19:42:59.823668 ip-10-0-143-252 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 05 19:42:59.823668 ip-10-0-143-252 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 05 19:42:59.823668 ip-10-0-143-252 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 05 19:42:59.824585 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.824494 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 05 19:42:59.827529 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827511 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 19:42:59.827529 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827527 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 19:42:59.827529 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827531 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827535 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827538 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827541 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827544 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827546 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827549 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827553 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827556 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827558 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827561 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827564 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827567 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827570 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827572 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827575 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827578 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827582 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827586 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827589 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 19:42:59.827631 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827592 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827595 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827598 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827601 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827604 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827607 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827609 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827612 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827615 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827618 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827621 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827624 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827626 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827629 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827632 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827635 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827639 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827642 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827645 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827647 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 19:42:59.828120 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827651 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827653 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827656 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827659 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827661 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827663 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827666 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827668 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827671 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827675 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827679 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827682 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827684 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827686 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827690 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827692 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827694 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827697 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827699 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 19:42:59.828607 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827702 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827704 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827707 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827710 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827713 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827715 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827718 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827722 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827724 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827727 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827730 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827733 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827735 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827739 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827742 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827745 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827748 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827750 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827753 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827758 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 19:42:59.829089 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827760 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827763 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827766 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827769 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827771 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827774 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827776 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827779 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827781 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827784 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827787 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827790 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827792 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827806 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827809 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827812 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827816 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.827818 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829432 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829440 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 19:42:59.829591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829444 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829447 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829450 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829455 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829459 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829462 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829465 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829468 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829471 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829473 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829476 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829479 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829482 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829485 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829487 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829490 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829493 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829496 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829498 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829501 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 19:42:59.830094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829503 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829506 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829509 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829511 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829514 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829516 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829519 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829522 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829525 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829527 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829530 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829532 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829535 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829537 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829540 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829542 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829545 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829548 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829550 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829554 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 19:42:59.830642 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829556 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829559 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829561 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829563 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829566 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829568 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829571 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829574 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829576 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829580 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829582 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829585 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829588 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829590 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829593 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829596 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829598 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829601 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829604 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 19:42:59.831163 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829606 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829609 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829612 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829615 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829617 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829620 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829622 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829624 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829627 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829629 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829632 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829636 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829638 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829641 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829643 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829646 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829648 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829650 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829653 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 19:42:59.831634 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829656 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829660 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829663 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829667 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829669 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829673 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829676 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829679 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829682 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829685 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829688 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829690 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829693 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829695 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829698 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829700 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829702 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829705 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.829708 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 19:42:59.832118 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830141 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830146 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830150 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830153 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830157 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830160 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830163 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830166 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830168 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830171 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830174 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830176 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830180 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830183 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830185 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830188 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830191 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830194 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830196 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830199 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 19:42:59.832588 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830202 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830204 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830207 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830210 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830213 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830215 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830218 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830220 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830223 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830225 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830228 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830231 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830233 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830236 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830240 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830243 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830246 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830249 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830252 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 19:42:59.833094 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830255 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830257 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830260 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830263 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830265 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830268 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830271 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830273 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830276 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830278 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830281 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830284 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830288 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830291 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830294 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830296 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830299 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830302 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830304 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 19:42:59.834224 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830308 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830311 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830316 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830320 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830323 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830326 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830328 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830331 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830334 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830336 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830338 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830341 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830344 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830346 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830349 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830351 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830354 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830356 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830358 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830361 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 19:42:59.834695 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830363 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830367 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830369 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830372 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830374 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830377 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830379 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830382 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830384 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830387 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830389 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830391 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830394 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830397 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830400 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830403 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830405 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830408 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830410 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830412 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 19:42:59.835202 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.830415 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830478 2578 flags.go:64] FLAG: --address="0.0.0.0" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830512 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830520 2578 flags.go:64] FLAG: --anonymous-auth="true" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830524 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830529 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830533 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830537 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830541 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830544 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830547 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830550 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830554 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830557 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830560 2578 flags.go:64] FLAG: --cgroup-root="" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830563 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830566 2578 flags.go:64] FLAG: --client-ca-file="" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830569 2578 flags.go:64] FLAG: --cloud-provider="external" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830572 2578 flags.go:64] FLAG: --cluster-dns="[]" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830576 2578 flags.go:64] FLAG: --cluster-domain="" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830578 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830582 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830585 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830588 2578 flags.go:64] FLAG: --container-log-max-files="5" Sep 05 19:42:59.835698 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830592 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830595 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830599 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830602 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830605 2578 flags.go:64] FLAG: --contention-profiling="false" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830608 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830611 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830614 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830617 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830621 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830624 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830626 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830629 2578 flags.go:64] FLAG: --enable-load-reader="false" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830632 2578 flags.go:64] FLAG: --enable-server="true" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830636 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830641 2578 flags.go:64] FLAG: --event-burst="100" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830644 2578 flags.go:64] FLAG: --event-qps="50" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830647 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830650 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830653 2578 flags.go:64] FLAG: --eviction-hard="" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830657 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830660 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830663 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830666 2578 flags.go:64] FLAG: --eviction-soft="" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830669 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 05 19:42:59.836344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830672 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830675 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830678 2578 flags.go:64] FLAG: --experimental-mounter-path="" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830680 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830683 2578 flags.go:64] FLAG: --fail-swap-on="true" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830686 2578 flags.go:64] FLAG: --feature-gates="" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830690 2578 flags.go:64] FLAG: --file-check-frequency="20s" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830692 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830696 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830700 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830703 2578 flags.go:64] FLAG: --healthz-port="10248" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830706 2578 flags.go:64] FLAG: --help="false" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830708 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-143-252.ec2.internal" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830711 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830714 2578 flags.go:64] FLAG: --http-check-frequency="20s" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830717 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830720 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830723 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830726 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830729 2578 flags.go:64] FLAG: --image-service-endpoint="" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830731 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830735 2578 flags.go:64] FLAG: --kube-api-burst="100" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830739 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830742 2578 flags.go:64] FLAG: --kube-api-qps="50" Sep 05 19:42:59.836976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830745 2578 flags.go:64] FLAG: --kube-reserved="" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830748 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830750 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830753 2578 flags.go:64] FLAG: --kubelet-cgroups="" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830756 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830759 2578 flags.go:64] FLAG: --lock-file="" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830761 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830764 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830767 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830772 2578 flags.go:64] FLAG: --log-json-split-stream="false" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830775 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830778 2578 flags.go:64] FLAG: --log-text-split-stream="false" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830780 2578 flags.go:64] FLAG: --logging-format="text" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830783 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830786 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830789 2578 flags.go:64] FLAG: --manifest-url="" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830792 2578 flags.go:64] FLAG: --manifest-url-header="" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830809 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830813 2578 flags.go:64] FLAG: --max-open-files="1000000" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830816 2578 flags.go:64] FLAG: --max-pods="110" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830819 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830822 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830825 2578 flags.go:64] FLAG: --memory-manager-policy="None" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830828 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830831 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 05 19:42:59.837751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830833 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830836 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830843 2578 flags.go:64] FLAG: --node-status-max-images="50" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830846 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830850 2578 flags.go:64] FLAG: --oom-score-adj="-999" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830853 2578 flags.go:64] FLAG: --pod-cidr="" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830856 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830862 2578 flags.go:64] FLAG: --pod-manifest-path="" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830864 2578 flags.go:64] FLAG: --pod-max-pids="-1" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830867 2578 flags.go:64] FLAG: --pods-per-core="0" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830870 2578 flags.go:64] FLAG: --port="10250" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830873 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830876 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01cbc7e75c6f3cda5" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830886 2578 flags.go:64] FLAG: --qos-reserved="" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830889 2578 flags.go:64] FLAG: --read-only-port="10255" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830892 2578 flags.go:64] FLAG: --register-node="true" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830896 2578 flags.go:64] FLAG: --register-with-taints="" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830899 2578 flags.go:64] FLAG: --registry-burst="10" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830902 2578 flags.go:64] FLAG: --registry-qps="5" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830905 2578 flags.go:64] FLAG: --reserved-cpus="" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830908 2578 flags.go:64] FLAG: --reserved-memory="" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830912 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830915 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830919 2578 flags.go:64] FLAG: --rotate-certificates="false" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830923 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 05 19:42:59.838967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830926 2578 flags.go:64] FLAG: --runonce="false" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830929 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830932 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830935 2578 flags.go:64] FLAG: --seccomp-default="false" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830937 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830941 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830944 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830947 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830950 2578 flags.go:64] FLAG: --storage-driver-password="root" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830953 2578 flags.go:64] FLAG: --storage-driver-secure="false" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830955 2578 flags.go:64] FLAG: --storage-driver-table="stats" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830958 2578 flags.go:64] FLAG: --storage-driver-user="root" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830962 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830965 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830969 2578 flags.go:64] FLAG: --system-cgroups="" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830972 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830977 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830980 2578 flags.go:64] FLAG: --tls-cert-file="" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830983 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830987 2578 flags.go:64] FLAG: --tls-min-version="" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830990 2578 flags.go:64] FLAG: --tls-private-key-file="" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830992 2578 flags.go:64] FLAG: --topology-manager-policy="none" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830995 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.830998 2578 flags.go:64] FLAG: --topology-manager-scope="container" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.831001 2578 flags.go:64] FLAG: --v="2" Sep 05 19:42:59.840058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.831005 2578 flags.go:64] FLAG: --version="false" Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.831009 2578 flags.go:64] FLAG: --vmodule="" Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.831013 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.831016 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831145 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831149 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831155 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831158 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831162 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831166 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831169 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831172 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831175 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831178 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831181 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831183 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831186 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831188 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831191 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831195 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 19:42:59.841168 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831197 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831200 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831203 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831206 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831208 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831211 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831213 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831215 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831218 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831221 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831223 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831226 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831228 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831231 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831233 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831236 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831239 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831242 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831245 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831248 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 19:42:59.841692 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831250 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831252 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831255 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831257 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831260 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831262 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831265 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831267 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831270 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831272 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831274 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831278 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831281 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831283 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831286 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831289 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831291 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831294 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831297 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831299 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 19:42:59.842715 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831302 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831304 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831306 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831309 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831312 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831314 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831317 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831319 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831322 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831324 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831327 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831330 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831333 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831335 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831338 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831340 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831343 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831346 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831348 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831351 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 19:42:59.843456 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831353 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831355 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831359 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831364 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831367 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831370 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831373 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831376 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831379 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831382 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831385 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831388 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831391 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831394 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831396 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831399 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831402 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831405 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831409 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 19:42:59.844106 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831412 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831414 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831417 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.831420 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.831886 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.839604 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.839625 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.839644 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.839652 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839736 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839744 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839751 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839756 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839760 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 19:42:59.844591 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839765 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839769 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839773 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839779 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839783 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839789 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839811 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839818 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839823 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839827 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839831 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839835 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839839 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839846 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839852 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839860 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839866 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839871 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839875 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 19:42:59.845006 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839880 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839884 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839888 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839893 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839897 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839902 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839907 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839912 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839916 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839920 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839924 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839928 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839933 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839937 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839942 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839946 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839950 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839955 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839959 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839963 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 19:42:59.845540 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839967 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839972 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839977 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839981 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839985 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839990 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839994 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.839998 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840003 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840007 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840011 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840015 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840019 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840023 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840027 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840032 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840036 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840041 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840046 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840050 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840054 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 19:42:59.846125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840058 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840062 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840066 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840070 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840073 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840078 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840081 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840085 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840089 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840093 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840098 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840102 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840106 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840111 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840116 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840120 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840124 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840128 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840132 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 19:42:59.846677 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840136 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840141 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840145 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840149 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840153 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840157 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840161 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840165 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840170 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840175 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840179 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840184 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840188 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840192 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840196 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 19:42:59.847261 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.840204 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840393 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840402 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840407 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840411 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840416 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840420 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840424 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840429 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840433 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840437 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840441 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840447 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840451 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840455 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840459 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840463 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840468 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840472 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 19:42:59.847897 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840476 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840480 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840484 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840488 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840492 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840496 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840501 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840505 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840510 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840514 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840518 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840525 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840531 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840536 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840540 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840544 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840548 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840553 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840557 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 19:42:59.848652 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840561 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840565 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840569 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840573 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840577 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840582 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840586 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840591 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840595 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840599 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840603 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840608 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840612 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840616 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840620 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840625 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840629 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840633 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840637 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840642 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 19:42:59.849412 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840646 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840650 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840654 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840658 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840662 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840666 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840671 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840675 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840679 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840683 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840687 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840691 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840695 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840699 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840703 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840708 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840712 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840716 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840721 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840725 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 19:42:59.849941 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840730 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840736 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840742 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840747 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840751 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840755 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840759 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840763 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840768 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840772 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840776 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840780 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840785 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840789 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840808 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840813 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840817 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840821 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840826 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 19:42:59.850598 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840830 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 19:42:59.851146 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840834 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 19:42:59.851146 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:42:59.840838 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 19:42:59.851146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.840846 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 05 19:42:59.851146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.841017 2578 server.go:962] "Client rotation is on, will bootstrap in background" Sep 05 19:42:59.851146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.843861 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 05 19:42:59.851146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.844929 2578 server.go:1019] "Starting client certificate rotation" Sep 05 19:42:59.851146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.845033 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 05 19:42:59.851146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.845107 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 05 19:42:59.851146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.847970 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 05 19:42:59.851385 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.851323 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 05 19:42:59.861250 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.861229 2578 log.go:25] "Validated CRI v1 runtime API" Sep 05 19:42:59.864109 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.864088 2578 log.go:25] "Validated CRI v1 image API" Sep 05 19:42:59.864923 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.864910 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 05 19:42:59.872908 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.872887 2578 fs.go:135] Filesystem UUIDs: map[65b455eb-2f1d-47fe-babf-da5ce0602426:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e1fbf6f2-7d8d-4eb2-8682-019420e8dd4a:/dev/nvme0n1p3] Sep 05 19:42:59.872978 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.872907 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 05 19:42:59.878049 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.877931 2578 manager.go:216] Machine: {Timestamp:2026-09-05 19:42:59.876574771 +0000 UTC m=+0.361875635 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3076452 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2de5136da401d5340364b7e852b23a SystemUUID:ec2de513-6da4-01d5-3403-64b7e852b23a BootID:6ca622b5-b26d-4e18-aabe-7bdd621fc713 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fc:4a:68:20:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fc:4a:68:20:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:87:62:ce:a9:da Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 05 19:42:59.878049 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.878045 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 05 19:42:59.878163 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.878130 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 05 19:42:59.880492 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.880465 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 05 19:42:59.880636 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.880494 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-252.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 05 19:42:59.881607 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.881593 2578 topology_manager.go:138] "Creating topology manager with none policy" Sep 05 19:42:59.881647 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.881610 2578 container_manager_linux.go:309] "Creating device plugin manager" Sep 05 19:42:59.881647 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.881619 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 05 19:42:59.882221 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.882210 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 05 19:42:59.883174 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.883159 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 05 19:42:59.883458 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.883447 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 05 19:42:59.887096 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.887083 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 05 19:42:59.887226 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.887216 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 05 19:42:59.887268 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.887250 2578 kubelet.go:486] "Attempting to sync node with API server" Sep 05 19:42:59.887299 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.887273 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 05 19:42:59.887299 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.887297 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 05 19:42:59.887353 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.887311 2578 kubelet.go:398] "Adding apiserver pod source" Sep 05 19:42:59.887353 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.887323 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 05 19:42:59.887764 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.887753 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 05 19:42:59.887842 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.887771 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 05 19:42:59.892439 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.892399 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 05 19:42:59.895159 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895141 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 05 19:42:59.895248 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895179 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 05 19:42:59.895348 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895337 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 05 19:42:59.895378 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895353 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 05 19:42:59.895378 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895359 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 05 19:42:59.895378 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895365 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 05 19:42:59.895378 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895373 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 05 19:42:59.895378 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895378 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 05 19:42:59.895938 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895927 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 05 19:42:59.895938 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895937 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 05 19:42:59.896009 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895945 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 05 19:42:59.896009 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895951 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 05 19:42:59.896009 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895960 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 05 19:42:59.896009 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.895967 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 05 19:42:59.896807 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.896786 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 05 19:42:59.896847 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.896809 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 05 19:42:59.899180 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.899163 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-252.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 05 19:42:59.899762 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:42:59.899739 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 05 19:42:59.899823 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:42:59.899746 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 05 19:42:59.900734 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.900722 2578 server.go:1268] "Started kubelet" Sep 05 19:42:59.900863 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.900821 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 05 19:42:59.900915 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.900832 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 05 19:42:59.900915 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.900903 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 05 19:42:59.901785 ip-10-0-143-252 systemd[1]: Started Kubernetes Kubelet. Sep 05 19:42:59.902176 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.902161 2578 server.go:310] "Adding debug handlers to kubelet server" Sep 05 19:42:59.902445 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.902404 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 05 19:42:59.907138 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.907045 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 05 19:42:59.907454 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.907426 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 05 19:42:59.908017 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:42:59.907998 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found" Sep 05 19:42:59.908589 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.908189 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 05 19:42:59.908589 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.908206 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 05 19:42:59.908589 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.908301 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 05 19:42:59.908589 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.908397 2578 reconstruct.go:97] "Volume reconstruction finished" Sep 05 19:42:59.908589 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.908404 2578 reconciler.go:29] "Reconciler: start to sync state" Sep 05 19:42:59.909170 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.909152 2578 factory.go:55] Registering systemd factory Sep 05 19:42:59.909170 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.909171 2578 factory.go:223] Registration of the systemd container factory successfully Sep 05 19:42:59.909663 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.909644 2578 factory.go:153] Registering CRI-O factory Sep 05 19:42:59.909663 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.909665 2578 factory.go:223] Registration of the crio container factory successfully Sep 05 19:42:59.909784 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.909716 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 05 19:42:59.909784 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.909747 2578 factory.go:103] Registering Raw factory Sep 05 19:42:59.909784 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.909764 2578 manager.go:1196] Started watching for new ooms in manager Sep 05 19:42:59.910316 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:42:59.910293 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 05 19:42:59.910969 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.910956 2578 manager.go:349] Starting recovery of all containers Sep 05 19:42:59.915408 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:42:59.915379 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 05 19:42:59.918810 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:42:59.918753 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 05 19:42:59.919499 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.919474 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 05 19:42:59.919747 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:42:59.918687 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-252.ec2.internal.18d284386cf748ef default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-252.ec2.internal,UID:ip-10-0-143-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-252.ec2.internal,},FirstTimestamp:2026-09-05 19:42:59.900688623 +0000 UTC m=+0.385989486,LastTimestamp:2026-09-05 19:42:59.900688623 +0000 UTC m=+0.385989486,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-252.ec2.internal,}" Sep 05 19:42:59.923559 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.923346 2578 manager.go:354] Recovery completed Sep 05 19:42:59.929707 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.929687 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 19:42:59.931597 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.931565 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory" Sep 05 19:42:59.931680 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.931611 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 19:42:59.931680 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.931622 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID" Sep 05 19:42:59.932220 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.932203 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 05 19:42:59.932220 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.932215 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 05 19:42:59.932346 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.932230 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 05 19:42:59.932346 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.932230 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 19:42:59.932960 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.932937 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory" Sep 05 19:42:59.933043 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.932974 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 19:42:59.933043 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.932988 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID" Sep 05 19:42:59.934129 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:42:59.934031 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-252.ec2.internal.18d284386ecedff3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-252.ec2.internal,UID:ip-10-0-143-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-252.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-252.ec2.internal,},FirstTimestamp:2026-09-05 19:42:59.931594739 +0000 UTC m=+0.416895602,LastTimestamp:2026-09-05 19:42:59.931594739 +0000 UTC m=+0.416895602,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-252.ec2.internal,}" Sep 05 19:42:59.934408 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.934394 2578 policy_none.go:49] "None policy: Start" Sep 05 19:42:59.934408 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.934410 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 05 19:42:59.934522 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.934420 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 05 19:42:59.936341 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.936328 2578 policy_none.go:47] "Start" Sep 05 19:42:59.936417 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.936399 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hbr95" Sep 05 19:42:59.945169 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:42:59.945092 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-252.ec2.internal.18d284386ecf3415 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-252.ec2.internal,UID:ip-10-0-143-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-252.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-252.ec2.internal,},FirstTimestamp:2026-09-05 19:42:59.931616277 +0000 UTC m=+0.416917141,LastTimestamp:2026-09-05 19:42:59.931616277 +0000 UTC m=+0.416917141,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-252.ec2.internal,}" Sep 05 19:42:59.946717 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.946698 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hbr95" Sep 05 19:42:59.971450 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.971430 2578 manager.go:337] "Starting Device Plugin manager" Sep 05 19:42:59.991493 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:42:59.971478 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 05 19:42:59.991493 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.971491 2578 server.go:87] "Starting device plugin registration server" Sep 05 19:42:59.991493 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.971890 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 05 19:42:59.991493 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.971904 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 05 19:42:59.991493 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.972320 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 05 19:42:59.991493 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.972410 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 05 19:42:59.991493 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:42:59.972419 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 05 19:42:59.991493 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:42:59.973791 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 05 19:42:59.991493 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:42:59.973857 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-252.ec2.internal\" not found" Sep 05 19:43:00.005591 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.005558 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 05 19:43:00.005740 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.005621 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 05 19:43:00.005740 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.005649 2578 kubelet.go:2448] "Starting kubelet main sync loop" Sep 05 19:43:00.005866 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.005762 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 05 19:43:00.008639 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.008616 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 19:43:00.073066 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.073035 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 19:43:00.074133 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.074059 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory" Sep 05 19:43:00.074133 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.074099 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 19:43:00.074133 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.074111 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID" Sep 05 19:43:00.074133 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.074133 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.085387 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.085368 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.085435 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.085392 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-252.ec2.internal\": node \"ip-10-0-143-252.ec2.internal\" not found" Sep 05 19:43:00.106881 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.106842 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal"] Sep 05 19:43:00.106950 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.106928 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 19:43:00.107787 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.107773 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory" Sep 05 19:43:00.107862 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.107819 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 19:43:00.107862 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.107831 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID" Sep 05 19:43:00.110119 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.110105 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 19:43:00.110227 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.110212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.110270 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.110243 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 19:43:00.110831 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.110811 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory" Sep 05 19:43:00.110939 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.110837 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 19:43:00.110939 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.110848 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID" Sep 05 19:43:00.110939 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.110812 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory" Sep 05 19:43:00.110939 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.110890 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 19:43:00.111087 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.110945 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID" Sep 05 19:43:00.111087 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.111076 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found" Sep 05 19:43:00.113085 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.113072 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.113143 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.113095 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 19:43:00.113830 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.113813 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory" Sep 05 19:43:00.113906 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.113843 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 19:43:00.113906 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.113853 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID" Sep 05 19:43:00.136561 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.136536 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-252.ec2.internal\" not found" node="ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.140956 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.140941 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-252.ec2.internal\" not found" node="ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.209273 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.209238 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bc8f954f2ebb5641595104f9f969e46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal\" (UID: \"0bc8f954f2ebb5641595104f9f969e46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.209273 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.209270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0bc8f954f2ebb5641595104f9f969e46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal\" (UID: \"0bc8f954f2ebb5641595104f9f969e46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.211459 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.211443 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found" Sep 05 19:43:00.309713 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.309672 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0bc8f954f2ebb5641595104f9f969e46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal\" (UID: \"0bc8f954f2ebb5641595104f9f969e46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.309713 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.309647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0bc8f954f2ebb5641595104f9f969e46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal\" (UID: \"0bc8f954f2ebb5641595104f9f969e46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.309907 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.309748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bc8f954f2ebb5641595104f9f969e46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal\" (UID: \"0bc8f954f2ebb5641595104f9f969e46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.309907 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.309770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dfffd2d51175b09bb92577df4eb5ea66-config\") pod \"kube-apiserver-proxy-ip-10-0-143-252.ec2.internal\" (UID: \"dfffd2d51175b09bb92577df4eb5ea66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.309907 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.309829 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bc8f954f2ebb5641595104f9f969e46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal\" (UID: \"0bc8f954f2ebb5641595104f9f969e46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.311762 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.311741 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found" Sep 05 19:43:00.410546 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.410483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dfffd2d51175b09bb92577df4eb5ea66-config\") pod \"kube-apiserver-proxy-ip-10-0-143-252.ec2.internal\" (UID: \"dfffd2d51175b09bb92577df4eb5ea66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.410546 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.410531 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dfffd2d51175b09bb92577df4eb5ea66-config\") pod \"kube-apiserver-proxy-ip-10-0-143-252.ec2.internal\" (UID: \"dfffd2d51175b09bb92577df4eb5ea66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.412593 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.412575 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found" Sep 05 19:43:00.439781 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.439753 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.444191 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.444172 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.513476 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.513432 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found" Sep 05 19:43:00.614348 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.614308 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found" Sep 05 19:43:00.715451 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.715351 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found" Sep 05 19:43:00.798707 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.798679 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 19:43:00.808252 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.808215 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.827987 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.827959 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 05 19:43:00.830421 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.830400 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal" Sep 05 19:43:00.843435 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.843410 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 05 19:43:00.844545 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.844515 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 05 19:43:00.844676 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.844657 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 05 19:43:00.844721 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.844693 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 05 19:43:00.887572 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.887542 2578 apiserver.go:52] "Watching apiserver" Sep 05 19:43:00.887769 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.887603 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 19:43:00.898909 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.898519 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 05 19:43:00.900515 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.900478 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5","openshift-image-registry/node-ca-qc2c8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal","openshift-multus/multus-additional-cni-plugins-rpglw","openshift-multus/multus-w7qgd","openshift-multus/network-metrics-daemon-89gvd","openshift-network-diagnostics/network-check-target-dsznw","kube-system/global-pull-secret-syncer-4l4q6","kube-system/konnectivity-agent-wcqth","openshift-cluster-node-tuning-operator/tuned-zgxrl","openshift-network-operator/iptables-alerter-zhwkf","openshift-ovn-kubernetes/ovnkube-node-kn472"] Sep 05 19:43:00.903266 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.903246 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wcqth" Sep 05 19:43:00.905636 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.905614 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qc2c8" Sep 05 19:43:00.906216 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.906198 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 05 19:43:00.906406 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.906390 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 05 19:43:00.906446 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.906411 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c2ggz\"" Sep 05 19:43:00.907247 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.907233 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 05 19:43:00.907995 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.907978 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:00.908553 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.908532 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 05 19:43:00.909050 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.909027 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 05 19:43:00.909351 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.909334 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 05 19:43:00.909599 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.909584 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jzh4k\"" Sep 05 19:43:00.910372 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.910355 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 05 19:43:00.910608 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.910594 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.911261 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.911244 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 05 19:43:00.911671 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.911657 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 05 19:43:00.911867 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.911846 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 05 19:43:00.912033 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.912020 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5vbt8\"" Sep 05 19:43:00.912223 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.912211 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 05 19:43:00.912963 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.912947 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:00.913029 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.913010 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:00.914640 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/452e1429-08ff-4b09-98b0-2a5f69bffa1a-agent-certs\") pod \"konnectivity-agent-wcqth\" (UID: \"452e1429-08ff-4b09-98b0-2a5f69bffa1a\") " pod="kube-system/konnectivity-agent-wcqth" Sep 05 19:43:00.914699 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51a28ad1-9a9b-47be-91b2-92dddc4a1408-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:00.914699 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914670 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51a28ad1-9a9b-47be-91b2-92dddc4a1408-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:00.914699 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914690 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-cnibin\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.914800 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-run-netns\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.914800 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-var-lib-cni-bin\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.914800 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914735 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-var-lib-cni-multus\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.914800 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914778 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-multus-conf-dir\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.914945 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8901e999-5de1-4671-8d64-7f40616f1667-cni-binary-copy\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.914945 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914846 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-run-multus-certs\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.914945 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914871 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/452e1429-08ff-4b09-98b0-2a5f69bffa1a-konnectivity-ca\") pod \"konnectivity-agent-wcqth\" (UID: \"452e1429-08ff-4b09-98b0-2a5f69bffa1a\") " pod="kube-system/konnectivity-agent-wcqth" Sep 05 19:43:00.914945 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-var-lib-kubelet\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.914945 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914920 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8901e999-5de1-4671-8d64-7f40616f1667-multus-daemon-config\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.915090 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b04d888b-736a-42e7-8a96-81d738b05abb-serviceca\") pod \"node-ca-qc2c8\" (UID: \"b04d888b-736a-42e7-8a96-81d738b05abb\") " pod="openshift-image-registry/node-ca-qc2c8" Sep 05 19:43:00.915090 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51a28ad1-9a9b-47be-91b2-92dddc4a1408-cni-binary-copy\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:00.915090 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.914997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-system-cni-dir\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.915090 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915023 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-multus-socket-dir-parent\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.915090 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-hostroot\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.915090 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bts2f\" (UniqueName: \"kubernetes.io/projected/51a28ad1-9a9b-47be-91b2-92dddc4a1408-kube-api-access-bts2f\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:00.915271 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915106 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-run-k8s-cni-cncf-io\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.915271 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b04d888b-736a-42e7-8a96-81d738b05abb-host\") pod \"node-ca-qc2c8\" (UID: \"b04d888b-736a-42e7-8a96-81d738b05abb\") " pod="openshift-image-registry/node-ca-qc2c8" Sep 05 19:43:00.915271 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51a28ad1-9a9b-47be-91b2-92dddc4a1408-system-cni-dir\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:00.915271 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcrz\" (UniqueName: \"kubernetes.io/projected/8901e999-5de1-4671-8d64-7f40616f1667-kube-api-access-zhcrz\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.915271 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915203 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxwpg\" (UniqueName: \"kubernetes.io/projected/b04d888b-736a-42e7-8a96-81d738b05abb-kube-api-access-nxwpg\") pod \"node-ca-qc2c8\" (UID: \"b04d888b-736a-42e7-8a96-81d738b05abb\") " pod="openshift-image-registry/node-ca-qc2c8" Sep 05 19:43:00.915271 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51a28ad1-9a9b-47be-91b2-92dddc4a1408-cnibin\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:00.915271 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51a28ad1-9a9b-47be-91b2-92dddc4a1408-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:00.915271 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915265 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-multus-cni-dir\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.915505 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915280 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-etc-kubernetes\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:00.915563 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.915549 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:00.915612 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.915598 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:00.917897 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.917880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:00.917948 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:00.917931 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:00.919634 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.919616 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:00.921248 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.921231 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 05 19:43:00.921887 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.921871 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8l74k\"" Sep 05 19:43:00.922101 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.922089 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zhwkf" Sep 05 19:43:00.922353 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.922332 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nxtk4\"" Sep 05 19:43:00.922935 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.922915 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 05 19:43:00.923033 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.922940 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 05 19:43:00.924664 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.924650 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:00.926304 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.926285 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 05 19:43:00.926389 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.926285 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 05 19:43:00.926436 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.926412 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 05 19:43:00.927113 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.927096 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:00.927238 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.927222 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rg6jh\"" Sep 05 19:43:00.927665 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.927648 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g4q8d\"" Sep 05 19:43:00.927713 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.927680 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 05 19:43:00.927915 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.927902 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 05 19:43:00.927956 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.927944 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 05 19:43:00.929267 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.929247 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 05 19:43:00.929362 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.929246 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 05 19:43:00.929876 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.929861 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 05 19:43:00.935657 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.935627 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 05 19:43:00.935755 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.935660 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 05 19:43:00.935755 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.935693 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 05 19:43:00.935962 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.935947 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m5n9z\"" Sep 05 19:43:00.937269 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.937253 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 05 19:43:00.951525 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.949886 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 19:38:00 +0000 UTC" deadline="2028-05-21 08:03:26.540145522 +0000 UTC" Sep 05 19:43:00.951525 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.949972 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14964h20m25.590178221s" Sep 05 19:43:00.954223 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.954205 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jscqw" Sep 05 19:43:00.968130 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:00.968065 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jscqw" Sep 05 19:43:01.009262 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.009242 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 05 19:43:01.015605 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-tuned\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.015665 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015618 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b04d888b-736a-42e7-8a96-81d738b05abb-serviceca\") pod \"node-ca-qc2c8\" (UID: \"b04d888b-736a-42e7-8a96-81d738b05abb\") " pod="openshift-image-registry/node-ca-qc2c8" Sep 05 19:43:01.015665 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015642 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:01.015737 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015690 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-run-systemd\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.015737 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-node-log\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.015737 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-system-cni-dir\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.015896 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-hostroot\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.015896 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a44efaa8-4af6-4534-8222-cf2bc6661930-iptables-alerter-script\") pod \"iptables-alerter-zhwkf\" (UID: \"a44efaa8-4af6-4534-8222-cf2bc6661930\") " pod="openshift-network-operator/iptables-alerter-zhwkf" Sep 05 19:43:01.015896 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-slash\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.015896 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015822 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-hostroot\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.015896 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-var-lib-openvswitch\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.016081 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015894 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-system-cni-dir\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.016081 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-ovnkube-script-lib\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.016081 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015952 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.016081 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.015984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-run-k8s-cni-cncf-io\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.016081 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a44efaa8-4af6-4534-8222-cf2bc6661930-host-slash\") pod \"iptables-alerter-zhwkf\" (UID: \"a44efaa8-4af6-4534-8222-cf2bc6661930\") " pod="openshift-network-operator/iptables-alerter-zhwkf" Sep 05 19:43:01.016081 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-kubelet\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.016081 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016044 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-run-k8s-cni-cncf-io\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.016081 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-ovn-node-metrics-cert\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.016350 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b04d888b-736a-42e7-8a96-81d738b05abb-host\") pod \"node-ca-qc2c8\" (UID: \"b04d888b-736a-42e7-8a96-81d738b05abb\") " pod="openshift-image-registry/node-ca-qc2c8" Sep 05 19:43:01.016350 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51a28ad1-9a9b-47be-91b2-92dddc4a1408-system-cni-dir\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.016350 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-run-netns\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.016350 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016155 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b04d888b-736a-42e7-8a96-81d738b05abb-host\") pod \"node-ca-qc2c8\" (UID: \"b04d888b-736a-42e7-8a96-81d738b05abb\") " pod="openshift-image-registry/node-ca-qc2c8" Sep 05 19:43:01.016350 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-socket-dir\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.016350 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016177 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51a28ad1-9a9b-47be-91b2-92dddc4a1408-system-cni-dir\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.016350 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-host\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.016350 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxwpg\" (UniqueName: \"kubernetes.io/projected/b04d888b-736a-42e7-8a96-81d738b05abb-kube-api-access-nxwpg\") pod \"node-ca-qc2c8\" (UID: \"b04d888b-736a-42e7-8a96-81d738b05abb\") " pod="openshift-image-registry/node-ca-qc2c8" Sep 05 19:43:01.016350 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51a28ad1-9a9b-47be-91b2-92dddc4a1408-cnibin\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.016350 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016332 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51a28ad1-9a9b-47be-91b2-92dddc4a1408-cnibin\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.016350 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016345 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-etc-openvswitch\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-systemd\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51a28ad1-9a9b-47be-91b2-92dddc4a1408-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51a28ad1-9a9b-47be-91b2-92dddc4a1408-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-cnibin\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-run-netns\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-var-lib-cni-bin\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016522 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-cnibin\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-run-multus-certs\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-run-ovn-kubernetes\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016559 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-run-multus-certs\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-cni-netd\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016586 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-run-netns\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016606 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-var-lib-cni-bin\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v62bw\" (UniqueName: \"kubernetes.io/projected/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-kube-api-access-v62bw\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-lib-modules\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.016725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016661 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b04d888b-736a-42e7-8a96-81d738b05abb-serviceca\") pod \"node-ca-qc2c8\" (UID: \"b04d888b-736a-42e7-8a96-81d738b05abb\") " pod="openshift-image-registry/node-ca-qc2c8" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/452e1429-08ff-4b09-98b0-2a5f69bffa1a-konnectivity-ca\") pod \"konnectivity-agent-wcqth\" (UID: \"452e1429-08ff-4b09-98b0-2a5f69bffa1a\") " pod="kube-system/konnectivity-agent-wcqth" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-var-lib-kubelet\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016748 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016763 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-var-lib-kubelet\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016775 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-device-dir\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016801 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m542x\" (UniqueName: \"kubernetes.io/projected/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-kube-api-access-m542x\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016842 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-sysconfig\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-sys\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8qh4\" (UniqueName: \"kubernetes.io/projected/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-kube-api-access-v8qh4\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016920 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51a28ad1-9a9b-47be-91b2-92dddc4a1408-cni-binary-copy\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-multus-socket-dir-parent\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42drz\" (UniqueName: \"kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz\") pod \"network-check-target-dsznw\" (UID: \"3940f127-ec8e-40e7-85b4-1545fe589b92\") " pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.016984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-systemd-units\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-run-ovn\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017020 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51a28ad1-9a9b-47be-91b2-92dddc4a1408-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-tmp\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.017460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017054 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-multus-socket-dir-parent\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017059 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51a28ad1-9a9b-47be-91b2-92dddc4a1408-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bts2f\" (UniqueName: \"kubernetes.io/projected/51a28ad1-9a9b-47be-91b2-92dddc4a1408-kube-api-access-bts2f\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017113 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-kubelet-config\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-ovnkube-config\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017188 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/452e1429-08ff-4b09-98b0-2a5f69bffa1a-konnectivity-ca\") pod \"konnectivity-agent-wcqth\" (UID: \"452e1429-08ff-4b09-98b0-2a5f69bffa1a\") " pod="kube-system/konnectivity-agent-wcqth" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017194 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-registration-dir\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-sysctl-conf\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcrz\" (UniqueName: \"kubernetes.io/projected/8901e999-5de1-4671-8d64-7f40616f1667-kube-api-access-zhcrz\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-env-overrides\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-sys-fs\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51a28ad1-9a9b-47be-91b2-92dddc4a1408-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017354 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51a28ad1-9a9b-47be-91b2-92dddc4a1408-cni-binary-copy\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-multus-cni-dir\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-etc-kubernetes\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017411 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-modprobe-d\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.018146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-sysctl-d\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-var-lib-kubelet\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017467 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-multus-cni-dir\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017485 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51a28ad1-9a9b-47be-91b2-92dddc4a1408-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017493 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-etc-kubernetes\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/452e1429-08ff-4b09-98b0-2a5f69bffa1a-agent-certs\") pod \"konnectivity-agent-wcqth\" (UID: \"452e1429-08ff-4b09-98b0-2a5f69bffa1a\") " pod="kube-system/konnectivity-agent-wcqth" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-var-lib-cni-multus\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-multus-conf-dir\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017619 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-host-var-lib-cni-multus\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8901e999-5de1-4671-8d64-7f40616f1667-cni-binary-copy\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017663 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8901e999-5de1-4671-8d64-7f40616f1667-multus-conf-dir\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-dbus\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017704 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-run-openvswitch\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-cni-bin\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8901e999-5de1-4671-8d64-7f40616f1667-multus-daemon-config\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qs8p\" (UniqueName: \"kubernetes.io/projected/a44efaa8-4af6-4534-8222-cf2bc6661930-kube-api-access-2qs8p\") pod \"iptables-alerter-zhwkf\" (UID: \"a44efaa8-4af6-4534-8222-cf2bc6661930\") " pod="openshift-network-operator/iptables-alerter-zhwkf" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017832 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-log-socket\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlc92\" (UniqueName: \"kubernetes.io/projected/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-kube-api-access-wlc92\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.018615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017919 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 05 19:43:01.019168 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-etc-selinux\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.019168 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-kubernetes\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.019168 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.017983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-run\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.019168 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.018110 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8901e999-5de1-4671-8d64-7f40616f1667-cni-binary-copy\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.019168 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.018199 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8901e999-5de1-4671-8d64-7f40616f1667-multus-daemon-config\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.021873 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.021828 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/452e1429-08ff-4b09-98b0-2a5f69bffa1a-agent-certs\") pod \"konnectivity-agent-wcqth\" (UID: \"452e1429-08ff-4b09-98b0-2a5f69bffa1a\") " pod="kube-system/konnectivity-agent-wcqth" Sep 05 19:43:01.032135 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.032110 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxwpg\" (UniqueName: \"kubernetes.io/projected/b04d888b-736a-42e7-8a96-81d738b05abb-kube-api-access-nxwpg\") pod \"node-ca-qc2c8\" (UID: \"b04d888b-736a-42e7-8a96-81d738b05abb\") " pod="openshift-image-registry/node-ca-qc2c8" Sep 05 19:43:01.033809 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.033787 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bts2f\" (UniqueName: \"kubernetes.io/projected/51a28ad1-9a9b-47be-91b2-92dddc4a1408-kube-api-access-bts2f\") pod \"multus-additional-cni-plugins-rpglw\" (UID: \"51a28ad1-9a9b-47be-91b2-92dddc4a1408\") " pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.034093 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.034075 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcrz\" (UniqueName: \"kubernetes.io/projected/8901e999-5de1-4671-8d64-7f40616f1667-kube-api-access-zhcrz\") pod \"multus-w7qgd\" (UID: \"8901e999-5de1-4671-8d64-7f40616f1667\") " pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.118387 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.118352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qs8p\" (UniqueName: \"kubernetes.io/projected/a44efaa8-4af6-4534-8222-cf2bc6661930-kube-api-access-2qs8p\") pod \"iptables-alerter-zhwkf\" (UID: \"a44efaa8-4af6-4534-8222-cf2bc6661930\") " pod="openshift-network-operator/iptables-alerter-zhwkf" Sep 05 19:43:01.118387 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.118395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-log-socket\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.118759 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.118423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlc92\" (UniqueName: \"kubernetes.io/projected/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-kube-api-access-wlc92\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.118859 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.118837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-etc-selinux\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.118935 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.118910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-kubernetes\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.118989 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.118941 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-log-socket\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.118989 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.118949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-run\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.119074 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-tuned\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.119128 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119089 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-kubernetes\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.119128 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119110 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-run\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.119232 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:01.119232 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119030 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-etc-selinux\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.119232 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-run-systemd\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.119232 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-node-log\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.119434 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.119239 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:01.119434 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a44efaa8-4af6-4534-8222-cf2bc6661930-iptables-alerter-script\") pod \"iptables-alerter-zhwkf\" (UID: \"a44efaa8-4af6-4534-8222-cf2bc6661930\") " pod="openshift-network-operator/iptables-alerter-zhwkf" Sep 05 19:43:01.119434 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-slash\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.119434 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-var-lib-openvswitch\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.119434 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-ovnkube-script-lib\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.119434 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.119739 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.119446 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret podName:1ff9d244-8eb8-4b0d-ad2d-618e28803d5a nodeName:}" failed. No retries permitted until 2026-09-05 19:43:01.619376609 +0000 UTC m=+2.104677479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret") pod "global-pull-secret-syncer-4l4q6" (UID: "1ff9d244-8eb8-4b0d-ad2d-618e28803d5a") : object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:01.119739 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119454 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.119739 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a44efaa8-4af6-4534-8222-cf2bc6661930-host-slash\") pod \"iptables-alerter-zhwkf\" (UID: \"a44efaa8-4af6-4534-8222-cf2bc6661930\") " pod="openshift-network-operator/iptables-alerter-zhwkf" Sep 05 19:43:01.119739 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119537 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-kubelet\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.119739 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119549 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-run-systemd\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.119739 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-ovn-node-metrics-cert\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.119739 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119603 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-slash\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.119739 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119639 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a44efaa8-4af6-4534-8222-cf2bc6661930-host-slash\") pod \"iptables-alerter-zhwkf\" (UID: \"a44efaa8-4af6-4534-8222-cf2bc6661930\") " pod="openshift-network-operator/iptables-alerter-zhwkf" Sep 05 19:43:01.119739 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119650 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-node-log\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.119739 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119712 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-kubelet\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.119739 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-run-netns\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.120324 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119778 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-var-lib-openvswitch\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.120324 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-socket-dir\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.120324 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119859 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-host\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.120324 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119951 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-run-netns\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.120324 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.119951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-etc-openvswitch\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.120580 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.120459 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a44efaa8-4af6-4534-8222-cf2bc6661930-iptables-alerter-script\") pod \"iptables-alerter-zhwkf\" (UID: \"a44efaa8-4af6-4534-8222-cf2bc6661930\") " pod="openshift-network-operator/iptables-alerter-zhwkf" Sep 05 19:43:01.120580 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.120492 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-ovnkube-script-lib\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.120754 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.120611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.120754 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.120656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-systemd\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.120754 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.120698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-run-ovn-kubernetes\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.120754 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.120729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-cni-netd\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.120994 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.120766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v62bw\" (UniqueName: \"kubernetes.io/projected/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-kube-api-access-v62bw\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:01.120994 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.120801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-lib-modules\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.120994 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.120838 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:01.120994 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.120868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-device-dir\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.120994 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.120925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m542x\" (UniqueName: \"kubernetes.io/projected/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-kube-api-access-m542x\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.120994 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.120976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-sysconfig\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.121307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-sys\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.121307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8qh4\" (UniqueName: \"kubernetes.io/projected/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-kube-api-access-v8qh4\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.121307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42drz\" (UniqueName: \"kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz\") pod \"network-check-target-dsznw\" (UID: \"3940f127-ec8e-40e7-85b4-1545fe589b92\") " pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:01.121307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-systemd-units\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.121307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-run-ovn\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.121307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-tmp\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.121307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-kubelet-config\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:01.121307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-ovnkube-config\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.121307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-registration-dir\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.121718 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-sysctl-conf\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.121718 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121333 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-device-dir\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.121718 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121383 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-env-overrides\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.121718 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-sys-fs\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.121718 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-modprobe-d\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.121718 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-sysctl-d\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.122058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121712 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-sysctl-d\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.122058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121835 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-sysconfig\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.122058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121899 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-socket-dir\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.122058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121947 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-kubelet-config\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:01.122058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.121967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-run-ovn\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.122058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122053 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-run-ovn-kubernetes\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.122369 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122121 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-sys\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.122430 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122397 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-cni-netd\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.122613 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122576 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-ovnkube-config\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.122703 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-var-lib-kubelet\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.122703 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122685 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-tuned\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.122813 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-dbus\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:01.122813 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-run-openvswitch\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.122813 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-cni-bin\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.122813 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122774 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-lib-modules\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.123120 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122865 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-registration-dir\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.123120 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122891 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-etc-openvswitch\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.123120 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.122947 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-var-lib-kubelet\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.123120 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.122973 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:01.123120 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.123035 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-systemd-units\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.123120 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.123036 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-run-openvswitch\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.123120 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.123092 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs podName:fe2aaad4-dc65-4f15-be9a-9fe0babba5b7 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:01.623056161 +0000 UTC m=+2.108357016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs") pod "network-metrics-daemon-89gvd" (UID: "fe2aaad4-dc65-4f15-be9a-9fe0babba5b7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:01.123120 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.123106 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-dbus\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:01.123120 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.123109 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-sys-fs\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.123576 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.123235 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-sysctl-conf\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.123576 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.123285 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-cni-bin\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.123576 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.123332 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-env-overrides\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.123576 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.123345 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-systemd\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.123576 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.123388 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.123576 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.123452 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-etc-modprobe-d\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.123576 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.123481 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-host\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.124542 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.124010 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-ovn-node-metrics-cert\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.125894 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.125847 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-tmp\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.134115 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.134082 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 19:43:01.134115 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.134114 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 19:43:01.134375 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.134131 2578 projected.go:196] Error preparing data for projected volume kube-api-access-42drz for pod openshift-network-diagnostics/network-check-target-dsznw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:01.134375 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.134201 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz podName:3940f127-ec8e-40e7-85b4-1545fe589b92 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:01.634179059 +0000 UTC m=+2.119479936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-42drz" (UniqueName: "kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz") pod "network-check-target-dsznw" (UID: "3940f127-ec8e-40e7-85b4-1545fe589b92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:01.134579 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.134552 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qs8p\" (UniqueName: \"kubernetes.io/projected/a44efaa8-4af6-4534-8222-cf2bc6661930-kube-api-access-2qs8p\") pod \"iptables-alerter-zhwkf\" (UID: \"a44efaa8-4af6-4534-8222-cf2bc6661930\") " pod="openshift-network-operator/iptables-alerter-zhwkf" Sep 05 19:43:01.134658 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.134551 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m542x\" (UniqueName: \"kubernetes.io/projected/73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79-kube-api-access-m542x\") pod \"aws-ebs-csi-driver-node-jfth5\" (UID: \"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.134658 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.134651 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlc92\" (UniqueName: \"kubernetes.io/projected/bfd09d5d-9d6d-4b49-b871-10b71cd7eff2-kube-api-access-wlc92\") pod \"ovnkube-node-kn472\" (UID: \"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.136471 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.136443 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8qh4\" (UniqueName: \"kubernetes.io/projected/9e2996a2-4df7-49c5-9217-f6dfbce3eaeb-kube-api-access-v8qh4\") pod \"tuned-zgxrl\" (UID: \"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb\") " pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.136649 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.136630 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v62bw\" (UniqueName: \"kubernetes.io/projected/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-kube-api-access-v62bw\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:01.233725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.233690 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wcqth" Sep 05 19:43:01.237137 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:43:01.237101 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfffd2d51175b09bb92577df4eb5ea66.slice/crio-5a7a591436573cd5aa7b800a2a034f2379f21b58c7612a038de97e45d68d3474 WatchSource:0}: Error finding container 5a7a591436573cd5aa7b800a2a034f2379f21b58c7612a038de97e45d68d3474: Status 404 returned error can't find the container with id 5a7a591436573cd5aa7b800a2a034f2379f21b58c7612a038de97e45d68d3474 Sep 05 19:43:01.243401 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.243382 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 19:43:01.250243 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:43:01.250216 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452e1429_08ff_4b09_98b0_2a5f69bffa1a.slice/crio-8f6f403aec63537111b9c102b357800ba44f09c62e42c0dbba8119f79592571c WatchSource:0}: Error finding container 8f6f403aec63537111b9c102b357800ba44f09c62e42c0dbba8119f79592571c: Status 404 returned error can't find the container with id 8f6f403aec63537111b9c102b357800ba44f09c62e42c0dbba8119f79592571c Sep 05 19:43:01.250719 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:43:01.250693 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc8f954f2ebb5641595104f9f969e46.slice/crio-7449f89cd906458d439fba92a496704ff69873caae8887af948568c644a6cebe WatchSource:0}: Error finding container 7449f89cd906458d439fba92a496704ff69873caae8887af948568c644a6cebe: Status 404 returned error can't find the container with id 7449f89cd906458d439fba92a496704ff69873caae8887af948568c644a6cebe Sep 05 19:43:01.250941 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.250919 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:01.259214 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.259192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qc2c8" Sep 05 19:43:01.263608 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:43:01.263583 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd09d5d_9d6d_4b49_b871_10b71cd7eff2.slice/crio-5ace38cdd474370028028831a6db3410b44203defacf9025df85443363fd5147 WatchSource:0}: Error finding container 5ace38cdd474370028028831a6db3410b44203defacf9025df85443363fd5147: Status 404 returned error can't find the container with id 5ace38cdd474370028028831a6db3410b44203defacf9025df85443363fd5147 Sep 05 19:43:01.268996 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:43:01.268969 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04d888b_736a_42e7_8a96_81d738b05abb.slice/crio-b6ced01d06999046c1096e607df620a363ef97c74689e961e82004d4fc0095bd WatchSource:0}: Error finding container b6ced01d06999046c1096e607df620a363ef97c74689e961e82004d4fc0095bd: Status 404 returned error can't find the container with id b6ced01d06999046c1096e607df620a363ef97c74689e961e82004d4fc0095bd Sep 05 19:43:01.275769 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.275748 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rpglw" Sep 05 19:43:01.286066 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:43:01.286045 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a28ad1_9a9b_47be_91b2_92dddc4a1408.slice/crio-1ee49ca6411f9aeb77c39bfda7aebf64b90872f3e9874bdcaddc3d311a13504e WatchSource:0}: Error finding container 1ee49ca6411f9aeb77c39bfda7aebf64b90872f3e9874bdcaddc3d311a13504e: Status 404 returned error can't find the container with id 1ee49ca6411f9aeb77c39bfda7aebf64b90872f3e9874bdcaddc3d311a13504e Sep 05 19:43:01.292068 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.292046 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w7qgd" Sep 05 19:43:01.296110 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.296093 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" Sep 05 19:43:01.300459 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:43:01.300438 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8901e999_5de1_4671_8d64_7f40616f1667.slice/crio-7dcf5db8af1bc80e3745723dc750202342bde5bfbcf9aae4b94499fefdce8b27 WatchSource:0}: Error finding container 7dcf5db8af1bc80e3745723dc750202342bde5bfbcf9aae4b94499fefdce8b27: Status 404 returned error can't find the container with id 7dcf5db8af1bc80e3745723dc750202342bde5bfbcf9aae4b94499fefdce8b27 Sep 05 19:43:01.305153 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:43:01.305130 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2996a2_4df7_49c5_9217_f6dfbce3eaeb.slice/crio-cd972edf5b5ff47d686be569c31a33fe99ead940d7833b3cee9b220922f8ed81 WatchSource:0}: Error finding container cd972edf5b5ff47d686be569c31a33fe99ead940d7833b3cee9b220922f8ed81: Status 404 returned error can't find the container with id cd972edf5b5ff47d686be569c31a33fe99ead940d7833b3cee9b220922f8ed81 Sep 05 19:43:01.313291 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.313272 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zhwkf" Sep 05 19:43:01.316724 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.316704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" Sep 05 19:43:01.324113 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:43:01.324078 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44efaa8_4af6_4534_8222_cf2bc6661930.slice/crio-f9e9a958ed40cb516758dbbc2def0ba008b6d062436b5a6f91a09654132dad9b WatchSource:0}: Error finding container f9e9a958ed40cb516758dbbc2def0ba008b6d062436b5a6f91a09654132dad9b: Status 404 returned error can't find the container with id f9e9a958ed40cb516758dbbc2def0ba008b6d062436b5a6f91a09654132dad9b Sep 05 19:43:01.326672 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:43:01.326650 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e5a37a_dc9d_4ba7_99f4_cbe7d7239f79.slice/crio-87dfc15a427db126cf93168b3113b05aa0413b2a9c4fc3068fecad940b16da59 WatchSource:0}: Error finding container 87dfc15a427db126cf93168b3113b05aa0413b2a9c4fc3068fecad940b16da59: Status 404 returned error can't find the container with id 87dfc15a427db126cf93168b3113b05aa0413b2a9c4fc3068fecad940b16da59 Sep 05 19:43:01.372627 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.372590 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 19:43:01.626683 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.626646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:01.626908 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.626693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:01.626908 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.626788 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:01.626908 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.626825 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:01.626908 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.626861 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs podName:fe2aaad4-dc65-4f15-be9a-9fe0babba5b7 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:02.626846717 +0000 UTC m=+3.112147583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs") pod "network-metrics-daemon-89gvd" (UID: "fe2aaad4-dc65-4f15-be9a-9fe0babba5b7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:01.626908 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.626890 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret podName:1ff9d244-8eb8-4b0d-ad2d-618e28803d5a nodeName:}" failed. No retries permitted until 2026-09-05 19:43:02.626872769 +0000 UTC m=+3.112173621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret") pod "global-pull-secret-syncer-4l4q6" (UID: "1ff9d244-8eb8-4b0d-ad2d-618e28803d5a") : object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:01.728330 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.728282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42drz\" (UniqueName: \"kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz\") pod \"network-check-target-dsznw\" (UID: \"3940f127-ec8e-40e7-85b4-1545fe589b92\") " pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:01.728558 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.728456 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 19:43:01.728558 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.728478 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 19:43:01.728558 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.728489 2578 projected.go:196] Error preparing data for projected volume kube-api-access-42drz for pod openshift-network-diagnostics/network-check-target-dsznw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:01.728558 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:01.728558 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz podName:3940f127-ec8e-40e7-85b4-1545fe589b92 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:02.728538121 +0000 UTC m=+3.213838983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-42drz" (UniqueName: "kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz") pod "network-check-target-dsznw" (UID: "3940f127-ec8e-40e7-85b4-1545fe589b92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:01.734260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.734230 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 19:43:01.971700 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.971601 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 19:38:01 +0000 UTC" deadline="2028-06-02 21:17:29.836404808 +0000 UTC" Sep 05 19:43:01.971700 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:01.971676 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15265h34m27.864759555s" Sep 05 19:43:02.043324 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.029243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" event={"ID":"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79","Type":"ContainerStarted","Data":"87dfc15a427db126cf93168b3113b05aa0413b2a9c4fc3068fecad940b16da59"} Sep 05 19:43:02.043324 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.036108 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpglw" event={"ID":"51a28ad1-9a9b-47be-91b2-92dddc4a1408","Type":"ContainerStarted","Data":"1ee49ca6411f9aeb77c39bfda7aebf64b90872f3e9874bdcaddc3d311a13504e"} Sep 05 19:43:02.047368 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.047170 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" event={"ID":"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2","Type":"ContainerStarted","Data":"5ace38cdd474370028028831a6db3410b44203defacf9025df85443363fd5147"} Sep 05 19:43:02.065844 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.065795 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wcqth" event={"ID":"452e1429-08ff-4b09-98b0-2a5f69bffa1a","Type":"ContainerStarted","Data":"8f6f403aec63537111b9c102b357800ba44f09c62e42c0dbba8119f79592571c"} Sep 05 19:43:02.074023 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.073906 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal" event={"ID":"dfffd2d51175b09bb92577df4eb5ea66","Type":"ContainerStarted","Data":"5a7a591436573cd5aa7b800a2a034f2379f21b58c7612a038de97e45d68d3474"} Sep 05 19:43:02.089667 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.089582 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zhwkf" event={"ID":"a44efaa8-4af6-4534-8222-cf2bc6661930","Type":"ContainerStarted","Data":"f9e9a958ed40cb516758dbbc2def0ba008b6d062436b5a6f91a09654132dad9b"} Sep 05 19:43:02.105480 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.105412 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" event={"ID":"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb","Type":"ContainerStarted","Data":"cd972edf5b5ff47d686be569c31a33fe99ead940d7833b3cee9b220922f8ed81"} Sep 05 19:43:02.117233 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.117191 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w7qgd" event={"ID":"8901e999-5de1-4671-8d64-7f40616f1667","Type":"ContainerStarted","Data":"7dcf5db8af1bc80e3745723dc750202342bde5bfbcf9aae4b94499fefdce8b27"} Sep 05 19:43:02.123415 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.123361 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qc2c8" event={"ID":"b04d888b-736a-42e7-8a96-81d738b05abb","Type":"ContainerStarted","Data":"b6ced01d06999046c1096e607df620a363ef97c74689e961e82004d4fc0095bd"} Sep 05 19:43:02.133004 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.132963 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" event={"ID":"0bc8f954f2ebb5641595104f9f969e46","Type":"ContainerStarted","Data":"7449f89cd906458d439fba92a496704ff69873caae8887af948568c644a6cebe"} Sep 05 19:43:02.635333 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.635290 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:02.635545 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.635382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:02.635545 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:02.635481 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:02.635545 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:02.635502 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:02.635721 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:02.635565 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs podName:fe2aaad4-dc65-4f15-be9a-9fe0babba5b7 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:04.635542717 +0000 UTC m=+5.120843595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs") pod "network-metrics-daemon-89gvd" (UID: "fe2aaad4-dc65-4f15-be9a-9fe0babba5b7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:02.635721 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:02.635588 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret podName:1ff9d244-8eb8-4b0d-ad2d-618e28803d5a nodeName:}" failed. No retries permitted until 2026-09-05 19:43:04.635578027 +0000 UTC m=+5.120878887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret") pod "global-pull-secret-syncer-4l4q6" (UID: "1ff9d244-8eb8-4b0d-ad2d-618e28803d5a") : object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:02.736415 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.736374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42drz\" (UniqueName: \"kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz\") pod \"network-check-target-dsznw\" (UID: \"3940f127-ec8e-40e7-85b4-1545fe589b92\") " pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:02.736594 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:02.736574 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 19:43:02.736674 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:02.736600 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 19:43:02.736674 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:02.736624 2578 projected.go:196] Error preparing data for projected volume kube-api-access-42drz for pod openshift-network-diagnostics/network-check-target-dsznw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:02.736787 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:02.736697 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz podName:3940f127-ec8e-40e7-85b4-1545fe589b92 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:04.736676412 +0000 UTC m=+5.221977271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-42drz" (UniqueName: "kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz") pod "network-check-target-dsznw" (UID: "3940f127-ec8e-40e7-85b4-1545fe589b92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:02.972599 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.972501 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 19:38:01 +0000 UTC" deadline="2028-05-17 14:34:07.244455612 +0000 UTC" Sep 05 19:43:02.972599 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:02.972545 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14874h51m4.271915519s" Sep 05 19:43:03.006213 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:03.006175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:03.006422 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:03.006346 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:03.006861 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:03.006837 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:03.007004 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:03.006980 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:03.007065 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:03.007053 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:03.007160 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:03.007142 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:03.448344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:03.448309 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 19:43:04.655439 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:04.655127 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:04.656045 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:04.655544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:04.656045 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:04.655410 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:04.656045 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:04.655802 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs podName:fe2aaad4-dc65-4f15-be9a-9fe0babba5b7 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:08.655777489 +0000 UTC m=+9.141078371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs") pod "network-metrics-daemon-89gvd" (UID: "fe2aaad4-dc65-4f15-be9a-9fe0babba5b7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:04.656884 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:04.655697 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:04.657050 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:04.656903 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret podName:1ff9d244-8eb8-4b0d-ad2d-618e28803d5a nodeName:}" failed. No retries permitted until 2026-09-05 19:43:08.656886123 +0000 UTC m=+9.142186981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret") pod "global-pull-secret-syncer-4l4q6" (UID: "1ff9d244-8eb8-4b0d-ad2d-618e28803d5a") : object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:04.757026 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:04.756985 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42drz\" (UniqueName: \"kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz\") pod \"network-check-target-dsznw\" (UID: \"3940f127-ec8e-40e7-85b4-1545fe589b92\") " pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:04.757246 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:04.757204 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 19:43:04.757246 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:04.757228 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 19:43:04.757246 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:04.757240 2578 projected.go:196] Error preparing data for projected volume kube-api-access-42drz for pod openshift-network-diagnostics/network-check-target-dsznw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:04.757416 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:04.757314 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz podName:3940f127-ec8e-40e7-85b4-1545fe589b92 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:08.757292935 +0000 UTC m=+9.242593811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-42drz" (UniqueName: "kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz") pod "network-check-target-dsznw" (UID: "3940f127-ec8e-40e7-85b4-1545fe589b92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:05.007006 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:05.006682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:05.007006 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:05.006733 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:05.007006 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:05.006882 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:05.007376 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:05.007328 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:05.007518 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:05.007485 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:05.007635 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:05.007577 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:06.836780 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.836734 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-26ptb"] Sep 05 19:43:06.840247 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.840206 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-26ptb" Sep 05 19:43:06.843523 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.843493 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ms42p\"" Sep 05 19:43:06.843781 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.843556 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 05 19:43:06.843781 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.843593 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 05 19:43:06.876723 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.876498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64v2n\" (UniqueName: \"kubernetes.io/projected/593dff6f-9c82-4da6-90cb-35032860e715-kube-api-access-64v2n\") pod \"node-resolver-26ptb\" (UID: \"593dff6f-9c82-4da6-90cb-35032860e715\") " pod="openshift-dns/node-resolver-26ptb" Sep 05 19:43:06.876723 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.876582 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/593dff6f-9c82-4da6-90cb-35032860e715-tmp-dir\") pod \"node-resolver-26ptb\" (UID: \"593dff6f-9c82-4da6-90cb-35032860e715\") " pod="openshift-dns/node-resolver-26ptb" Sep 05 19:43:06.876723 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.876617 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/593dff6f-9c82-4da6-90cb-35032860e715-hosts-file\") pod \"node-resolver-26ptb\" (UID: \"593dff6f-9c82-4da6-90cb-35032860e715\") " pod="openshift-dns/node-resolver-26ptb" Sep 05 19:43:06.977549 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.977489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64v2n\" (UniqueName: \"kubernetes.io/projected/593dff6f-9c82-4da6-90cb-35032860e715-kube-api-access-64v2n\") pod \"node-resolver-26ptb\" (UID: \"593dff6f-9c82-4da6-90cb-35032860e715\") " pod="openshift-dns/node-resolver-26ptb" Sep 05 19:43:06.977735 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.977580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/593dff6f-9c82-4da6-90cb-35032860e715-tmp-dir\") pod \"node-resolver-26ptb\" (UID: \"593dff6f-9c82-4da6-90cb-35032860e715\") " pod="openshift-dns/node-resolver-26ptb" Sep 05 19:43:06.977735 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.977616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/593dff6f-9c82-4da6-90cb-35032860e715-hosts-file\") pod \"node-resolver-26ptb\" (UID: \"593dff6f-9c82-4da6-90cb-35032860e715\") " pod="openshift-dns/node-resolver-26ptb" Sep 05 19:43:06.977735 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.977726 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/593dff6f-9c82-4da6-90cb-35032860e715-hosts-file\") pod \"node-resolver-26ptb\" (UID: \"593dff6f-9c82-4da6-90cb-35032860e715\") " pod="openshift-dns/node-resolver-26ptb" Sep 05 19:43:06.978380 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.978358 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/593dff6f-9c82-4da6-90cb-35032860e715-tmp-dir\") pod \"node-resolver-26ptb\" (UID: \"593dff6f-9c82-4da6-90cb-35032860e715\") " pod="openshift-dns/node-resolver-26ptb" Sep 05 19:43:06.989843 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:06.989809 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64v2n\" (UniqueName: \"kubernetes.io/projected/593dff6f-9c82-4da6-90cb-35032860e715-kube-api-access-64v2n\") pod \"node-resolver-26ptb\" (UID: \"593dff6f-9c82-4da6-90cb-35032860e715\") " pod="openshift-dns/node-resolver-26ptb" Sep 05 19:43:07.006184 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:07.006147 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:07.006184 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:07.006184 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:07.006419 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:07.006185 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:07.006419 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:07.006346 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:07.006514 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:07.006485 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:07.006600 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:07.006577 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:07.154794 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:07.154707 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-26ptb" Sep 05 19:43:08.692950 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:08.692614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:08.692950 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:08.692700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:08.692950 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:08.692848 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:08.692950 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:08.692852 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:08.692950 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:08.692923 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret podName:1ff9d244-8eb8-4b0d-ad2d-618e28803d5a nodeName:}" failed. No retries permitted until 2026-09-05 19:43:16.69290307 +0000 UTC m=+17.178203944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret") pod "global-pull-secret-syncer-4l4q6" (UID: "1ff9d244-8eb8-4b0d-ad2d-618e28803d5a") : object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:08.692950 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:08.692945 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs podName:fe2aaad4-dc65-4f15-be9a-9fe0babba5b7 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:16.692934108 +0000 UTC m=+17.178234959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs") pod "network-metrics-daemon-89gvd" (UID: "fe2aaad4-dc65-4f15-be9a-9fe0babba5b7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:08.793750 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:08.793057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42drz\" (UniqueName: \"kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz\") pod \"network-check-target-dsznw\" (UID: \"3940f127-ec8e-40e7-85b4-1545fe589b92\") " pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:08.793750 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:08.793270 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 19:43:08.793750 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:08.793293 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 19:43:08.793750 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:08.793306 2578 projected.go:196] Error preparing data for projected volume kube-api-access-42drz for pod openshift-network-diagnostics/network-check-target-dsznw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:08.793750 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:08.793373 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz podName:3940f127-ec8e-40e7-85b4-1545fe589b92 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:16.793354557 +0000 UTC m=+17.278655430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-42drz" (UniqueName: "kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz") pod "network-check-target-dsznw" (UID: "3940f127-ec8e-40e7-85b4-1545fe589b92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:09.006117 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:09.006082 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:09.006117 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:09.006112 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:09.006381 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:09.006082 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:09.006381 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:09.006217 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:09.006381 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:09.006296 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:09.006381 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:09.006353 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:11.006016 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:11.005970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:11.006016 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:11.006003 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:11.006016 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:11.006011 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:11.006527 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:11.006112 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:11.006527 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:11.006191 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:11.006527 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:11.006286 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:13.006912 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:13.006873 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:13.006912 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:13.006897 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:13.007450 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:13.006889 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:13.007450 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:13.007002 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:13.007450 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:13.007089 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:13.007450 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:13.007155 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:15.006177 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:15.006139 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:15.006177 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:15.006177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:15.006701 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:15.006139 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:15.006701 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:15.006252 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:15.006701 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:15.006355 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:15.006701 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:15.006436 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:16.748893 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:16.748853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:16.749320 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:16.748936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:16.749320 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:16.749016 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:16.749320 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:16.749075 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:16.749320 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:16.749100 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret podName:1ff9d244-8eb8-4b0d-ad2d-618e28803d5a nodeName:}" failed. No retries permitted until 2026-09-05 19:43:32.74907957 +0000 UTC m=+33.234380428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret") pod "global-pull-secret-syncer-4l4q6" (UID: "1ff9d244-8eb8-4b0d-ad2d-618e28803d5a") : object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:16.749320 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:16.749124 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs podName:fe2aaad4-dc65-4f15-be9a-9fe0babba5b7 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:32.749109908 +0000 UTC m=+33.234410763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs") pod "network-metrics-daemon-89gvd" (UID: "fe2aaad4-dc65-4f15-be9a-9fe0babba5b7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:16.849712 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:16.849618 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42drz\" (UniqueName: \"kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz\") pod \"network-check-target-dsznw\" (UID: \"3940f127-ec8e-40e7-85b4-1545fe589b92\") " pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:16.849910 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:16.849818 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 19:43:16.849910 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:16.849850 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 19:43:16.849910 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:16.849863 2578 projected.go:196] Error preparing data for projected volume kube-api-access-42drz for pod openshift-network-diagnostics/network-check-target-dsznw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:16.850070 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:16.849918 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz podName:3940f127-ec8e-40e7-85b4-1545fe589b92 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:32.849903251 +0000 UTC m=+33.335204106 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-42drz" (UniqueName: "kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz") pod "network-check-target-dsznw" (UID: "3940f127-ec8e-40e7-85b4-1545fe589b92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:17.006227 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:17.006190 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:17.006415 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:17.006191 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:17.006415 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:17.006316 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:17.006533 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:17.006203 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:17.006533 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:17.006428 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:17.006621 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:17.006526 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:19.006788 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:19.006750 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:19.007271 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:19.006808 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:19.007271 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:19.006752 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:19.007271 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:19.006899 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:19.007271 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:19.006994 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:19.007271 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:19.007045 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:21.006984 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.006668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:21.007755 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.006710 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:21.007755 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:21.007108 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:21.007755 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:21.007236 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:21.007755 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.006779 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:21.007755 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:21.007350 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:21.189669 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.189636 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpglw" event={"ID":"51a28ad1-9a9b-47be-91b2-92dddc4a1408","Type":"ContainerStarted","Data":"a2de85061f8408c7420e9c00aafeb274ac81c5b6a6a38f880a7bb37d6c3e1550"} Sep 05 19:43:21.192494 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.192473 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 19:43:21.192833 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.192813 2578 generic.go:356] "Generic (PLEG): container finished" podID="bfd09d5d-9d6d-4b49-b871-10b71cd7eff2" containerID="7a3b583133c6b33ecdfd92af6456b5a9c58f03f41f963f9400848c2a1292057c" exitCode=1 Sep 05 19:43:21.192904 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.192840 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" event={"ID":"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2","Type":"ContainerStarted","Data":"8f19710d18927428835713021af9fa3eee76605332de522c8b5b4ce29407c82a"} Sep 05 19:43:21.192953 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.192910 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" event={"ID":"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2","Type":"ContainerStarted","Data":"38912bbf67e233ce46a154e38bb9e1c74561627edcd2d72a8c719a751e80a9be"} Sep 05 19:43:21.192953 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.192947 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" event={"ID":"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2","Type":"ContainerStarted","Data":"71474e5435b5a604433361d7f19d0679cdefabbe12eee2ff9fe04898fa95f680"} Sep 05 19:43:21.193023 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.192988 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" event={"ID":"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2","Type":"ContainerStarted","Data":"752892517b7bbf9a5866e492e0671a81206d7b76fb6a03c4fb08deefe7ff9fee"} Sep 05 19:43:21.193023 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.193019 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" event={"ID":"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2","Type":"ContainerDied","Data":"7a3b583133c6b33ecdfd92af6456b5a9c58f03f41f963f9400848c2a1292057c"} Sep 05 19:43:21.193097 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.193050 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" event={"ID":"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2","Type":"ContainerStarted","Data":"6f9d6db08d82a27173e57c75c7aea7fe70d98a0ae7824361af5a95c5b44003e8"} Sep 05 19:43:21.197460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.197430 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wcqth" event={"ID":"452e1429-08ff-4b09-98b0-2a5f69bffa1a","Type":"ContainerStarted","Data":"969b2418792994ab9fd290d1051b4c72d35d116d136f6c7ace92a4e9a841810e"} Sep 05 19:43:21.198949 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.198914 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal" event={"ID":"dfffd2d51175b09bb92577df4eb5ea66","Type":"ContainerStarted","Data":"bb597cb7df50e99d73d66fc98313e2b23a9d8263bbd2c9e475cd7b25dec5a5dc"} Sep 05 19:43:21.200319 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.200298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-26ptb" event={"ID":"593dff6f-9c82-4da6-90cb-35032860e715","Type":"ContainerStarted","Data":"23d9de5f3139d01d2eab2b267b9d743d2d125e39d6f72db2dbee3e414a8ae150"} Sep 05 19:43:21.200415 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.200335 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-26ptb" event={"ID":"593dff6f-9c82-4da6-90cb-35032860e715","Type":"ContainerStarted","Data":"5903ca67406e756e02a897afd30b841adf662cf1184fadf09ba8f6ddfdfe3fbb"} Sep 05 19:43:21.201914 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.201892 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" event={"ID":"9e2996a2-4df7-49c5-9217-f6dfbce3eaeb","Type":"ContainerStarted","Data":"1dd00eab9cdc5a465337cd5c81bb04fbae2c56eb3261e8eeac9fe5e420ff9867"} Sep 05 19:43:21.203511 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.203489 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w7qgd" event={"ID":"8901e999-5de1-4671-8d64-7f40616f1667","Type":"ContainerStarted","Data":"c57814e8c0cc49f1a92d7c60afd6b304cda2cb42a22278f12aacc912e12f36c5"} Sep 05 19:43:21.205101 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.205017 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qc2c8" event={"ID":"b04d888b-736a-42e7-8a96-81d738b05abb","Type":"ContainerStarted","Data":"69a4fcf573f5e9b56d5b8bcd7ad26d893d4ed4a89d0944969c254b270b5751ac"} Sep 05 19:43:21.206779 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.206695 2578 generic.go:356] "Generic (PLEG): container finished" podID="0bc8f954f2ebb5641595104f9f969e46" containerID="01de6ab43a68a1d2c5724e4732527dc0143a04766df69af777926ab8687d533a" exitCode=0 Sep 05 19:43:21.206779 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.206763 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" event={"ID":"0bc8f954f2ebb5641595104f9f969e46","Type":"ContainerDied","Data":"01de6ab43a68a1d2c5724e4732527dc0143a04766df69af777926ab8687d533a"} Sep 05 19:43:21.208383 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.208363 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" event={"ID":"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79","Type":"ContainerStarted","Data":"1b0d8a362e6bb631c01b6135c20c8c109447fcc1a0901105f117375ea2ac47c6"} Sep 05 19:43:21.227073 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.227022 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-26ptb" podStartSLOduration=15.22700709 podStartE2EDuration="15.22700709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:21.226382708 +0000 UTC m=+21.711683581" watchObservedRunningTime="2026-09-05 19:43:21.22700709 +0000 UTC m=+21.712307963" Sep 05 19:43:21.273963 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.273873 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal" podStartSLOduration=20.273859271 podStartE2EDuration="20.273859271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:21.260475316 +0000 UTC m=+21.745776189" watchObservedRunningTime="2026-09-05 19:43:21.273859271 +0000 UTC m=+21.759160202" Sep 05 19:43:21.274138 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.274035 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qc2c8" podStartSLOduration=2.127877154 podStartE2EDuration="21.274028333s" totalImagesPullingTime="19.146151179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:21.273262166 +0000 UTC m=+21.758563050" watchObservedRunningTime="2026-09-05 19:43:21.274028333 +0000 UTC m=+21.759329208" Sep 05 19:43:21.289619 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.289564 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zgxrl" podStartSLOduration=2.179038576 podStartE2EDuration="21.289550866s" totalImagesPullingTime="19.11051229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:21.289188456 +0000 UTC m=+21.774489330" watchObservedRunningTime="2026-09-05 19:43:21.289550866 +0000 UTC m=+21.774851739" Sep 05 19:43:21.307975 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.307919 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wcqth" podStartSLOduration=2.144975897 podStartE2EDuration="21.307899404s" totalImagesPullingTime="19.162923507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:21.307864903 +0000 UTC m=+21.793165777" watchObservedRunningTime="2026-09-05 19:43:21.307899404 +0000 UTC m=+21.793200274" Sep 05 19:43:21.332169 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:21.332118 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w7qgd" podStartSLOduration=1.916379412 podStartE2EDuration="21.332101149s" totalImagesPullingTime="19.415721737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:21.330274533 +0000 UTC m=+21.815575406" watchObservedRunningTime="2026-09-05 19:43:21.332101149 +0000 UTC m=+21.817402022" Sep 05 19:43:22.187244 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.187219 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 05 19:43:22.212394 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.212354 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zhwkf" event={"ID":"a44efaa8-4af6-4534-8222-cf2bc6661930","Type":"ContainerStarted","Data":"d83d4d65c409d03dc05ca8462c83f47cf6adba6ce93027c004dd95dd55c13163"} Sep 05 19:43:22.214586 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.214525 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" event={"ID":"0bc8f954f2ebb5641595104f9f969e46","Type":"ContainerStarted","Data":"903962b1e33272b320bd2837ae978ec3731fd22d74b389fefa22effaeb8f8515"} Sep 05 19:43:22.217060 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.217031 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" event={"ID":"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79","Type":"ContainerStarted","Data":"5095627adafb1ece1fd388e260a6061dabd8a3bfe41aad41b95565fc711495b4"} Sep 05 19:43:22.218537 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.218513 2578 generic.go:356] "Generic (PLEG): container finished" podID="51a28ad1-9a9b-47be-91b2-92dddc4a1408" containerID="a2de85061f8408c7420e9c00aafeb274ac81c5b6a6a38f880a7bb37d6c3e1550" exitCode=0 Sep 05 19:43:22.218677 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.218597 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpglw" event={"ID":"51a28ad1-9a9b-47be-91b2-92dddc4a1408","Type":"ContainerDied","Data":"a2de85061f8408c7420e9c00aafeb274ac81c5b6a6a38f880a7bb37d6c3e1550"} Sep 05 19:43:22.227295 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.227257 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zhwkf" podStartSLOduration=3.149999679 podStartE2EDuration="22.227243655s" totalImagesPullingTime="19.077243976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:22.227234704 +0000 UTC m=+22.712535588" watchObservedRunningTime="2026-09-05 19:43:22.227243655 +0000 UTC m=+22.712544528" Sep 05 19:43:22.269661 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.269608 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" podStartSLOduration=21.26959289 podStartE2EDuration="21.26959289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:22.269570168 +0000 UTC m=+22.754871041" watchObservedRunningTime="2026-09-05 19:43:22.26959289 +0000 UTC m=+22.754893762" Sep 05 19:43:22.582121 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.582087 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wcqth" Sep 05 19:43:22.582840 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.582823 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wcqth" Sep 05 19:43:22.988335 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.988174 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T19:43:22.187238339Z","UUID":"0adf7cc9-bd12-4a1c-8108-a4ae72746bd4","Handler":null,"Name":"","Endpoint":""} Sep 05 19:43:22.989743 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.989719 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 05 19:43:22.989889 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:22.989752 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 05 19:43:23.006812 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:23.006761 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:23.007007 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:23.006763 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:23.007007 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:23.006893 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:23.007137 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:23.007005 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:23.007137 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:23.006774 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:23.007137 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:23.007125 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:23.224056 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:23.224023 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" event={"ID":"73e5a37a-dc9d-4ba7-99f4-cbe7d7239f79","Type":"ContainerStarted","Data":"a6ac17fee9b0dae89e81a22416fe6cdb50f07b05da50eb649af35a161a6f96ab"} Sep 05 19:43:24.229846 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:24.229629 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 19:43:24.230310 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:24.230277 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" event={"ID":"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2","Type":"ContainerStarted","Data":"1c841105b49b95c59ff70569ff0d86ebdddc35ecb084c6c2161f6275b6c6ebaf"} Sep 05 19:43:24.230310 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:24.230304 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 05 19:43:25.006530 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:25.006498 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:25.006530 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:25.006516 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:25.006839 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:25.006640 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:25.006839 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:25.006662 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:25.006839 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:25.006749 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:25.007014 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:25.006856 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:26.239171 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:26.239140 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 19:43:26.239699 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:26.239548 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" event={"ID":"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2","Type":"ContainerStarted","Data":"b6c23399ff25ce7e275bba1ae5b7595f666b0eb82f2e943ea08e4762b50b79ad"} Sep 05 19:43:26.240064 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:26.240028 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:26.240204 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:26.240090 2578 scope.go:117] "RemoveContainer" containerID="7a3b583133c6b33ecdfd92af6456b5a9c58f03f41f963f9400848c2a1292057c" Sep 05 19:43:26.240204 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:26.240099 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:26.256392 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:26.256366 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:26.256712 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:26.256696 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:26.269179 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:26.269132 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfth5" podStartSLOduration=4.306674952 podStartE2EDuration="25.269117954s" totalImagesPullingTime="20.962443002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:23.247607717 +0000 UTC m=+23.732908590" watchObservedRunningTime="2026-09-05 19:43:26.269117954 +0000 UTC m=+26.754418827" Sep 05 19:43:27.005942 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:27.005902 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:27.006094 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:27.006038 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:27.006227 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:27.006151 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:27.006227 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:27.006032 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:27.006227 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:27.006037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:27.006430 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:27.006285 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:27.296256 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:27.296215 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:43:27.398975 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:27.398945 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dsznw"] Sep 05 19:43:27.399083 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:27.399058 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:27.399186 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:27.399161 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:27.399630 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:27.399612 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4l4q6"] Sep 05 19:43:27.399690 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:27.399679 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:27.399764 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:27.399748 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:27.404165 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:27.404142 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-89gvd"] Sep 05 19:43:27.404282 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:27.404236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:27.404376 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:27.404356 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:28.245840 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:28.245790 2578 generic.go:356] "Generic (PLEG): container finished" podID="51a28ad1-9a9b-47be-91b2-92dddc4a1408" containerID="affe1e39bef48ea0f87c371d1a2bfa75487eda5c85091f70ba0e754cb7e8d664" exitCode=0 Sep 05 19:43:28.246098 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:28.245865 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpglw" event={"ID":"51a28ad1-9a9b-47be-91b2-92dddc4a1408","Type":"ContainerDied","Data":"affe1e39bef48ea0f87c371d1a2bfa75487eda5c85091f70ba0e754cb7e8d664"} Sep 05 19:43:28.249985 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:28.249968 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 19:43:28.250298 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:28.250280 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" event={"ID":"bfd09d5d-9d6d-4b49-b871-10b71cd7eff2","Type":"ContainerStarted","Data":"b57caa80b6470d56dd73b582a2149e42448e941b0872da83043b925bd8c6bc41"} Sep 05 19:43:28.299275 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:28.299225 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" podStartSLOduration=9.11412757 podStartE2EDuration="28.299208884s" totalImagesPullingTime="19.185081314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:28.299032989 +0000 UTC m=+28.784333862" watchObservedRunningTime="2026-09-05 19:43:28.299208884 +0000 UTC m=+28.784509757" Sep 05 19:43:29.003051 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:29.003011 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wcqth" Sep 05 19:43:29.003232 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:29.003135 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 05 19:43:29.004355 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:29.004013 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wcqth" Sep 05 19:43:29.006452 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:29.006328 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:29.006857 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:29.006839 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:29.006857 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:29.006847 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:29.007009 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:29.006882 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:29.007009 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:29.006958 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:29.007106 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:29.007045 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:30.255909 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:30.255704 2578 generic.go:356] "Generic (PLEG): container finished" podID="51a28ad1-9a9b-47be-91b2-92dddc4a1408" containerID="0c5a55fdb56a143c8a5634d2c2cf5939c48d34a22b45893b49d5119460b92b1a" exitCode=0 Sep 05 19:43:30.255909 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:30.255786 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpglw" event={"ID":"51a28ad1-9a9b-47be-91b2-92dddc4a1408","Type":"ContainerDied","Data":"0c5a55fdb56a143c8a5634d2c2cf5939c48d34a22b45893b49d5119460b92b1a"} Sep 05 19:43:31.006005 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:31.005976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:31.006131 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:31.005976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:31.006131 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:31.006100 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:31.006226 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:31.005976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:31.006226 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:31.006210 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:31.006309 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:31.006260 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:31.260989 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:31.260950 2578 generic.go:356] "Generic (PLEG): container finished" podID="51a28ad1-9a9b-47be-91b2-92dddc4a1408" containerID="21024263738b524780f2c040c395c2da843477817227305d4e97ba56537a43af" exitCode=0 Sep 05 19:43:31.261406 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:31.261022 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpglw" event={"ID":"51a28ad1-9a9b-47be-91b2-92dddc4a1408","Type":"ContainerDied","Data":"21024263738b524780f2c040c395c2da843477817227305d4e97ba56537a43af"} Sep 05 19:43:32.761886 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:32.761850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:32.762276 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:32.761909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:32.762276 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:32.761997 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:32.762276 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:32.761999 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:32.762276 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:32.762050 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret podName:1ff9d244-8eb8-4b0d-ad2d-618e28803d5a nodeName:}" failed. No retries permitted until 2026-09-05 19:44:04.76203571 +0000 UTC m=+65.247336560 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret") pod "global-pull-secret-syncer-4l4q6" (UID: "1ff9d244-8eb8-4b0d-ad2d-618e28803d5a") : object "kube-system"/"original-pull-secret" not registered Sep 05 19:43:32.762276 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:32.762064 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs podName:fe2aaad4-dc65-4f15-be9a-9fe0babba5b7 nodeName:}" failed. No retries permitted until 2026-09-05 19:44:04.762058071 +0000 UTC m=+65.247358922 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs") pod "network-metrics-daemon-89gvd" (UID: "fe2aaad4-dc65-4f15-be9a-9fe0babba5b7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 19:43:32.863139 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:32.863099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42drz\" (UniqueName: \"kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz\") pod \"network-check-target-dsznw\" (UID: \"3940f127-ec8e-40e7-85b4-1545fe589b92\") " pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:32.863330 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:32.863310 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 19:43:32.863405 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:32.863351 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 19:43:32.863405 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:32.863364 2578 projected.go:196] Error preparing data for projected volume kube-api-access-42drz for pod openshift-network-diagnostics/network-check-target-dsznw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:32.863506 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:32.863431 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz podName:3940f127-ec8e-40e7-85b4-1545fe589b92 nodeName:}" failed. No retries permitted until 2026-09-05 19:44:04.863410809 +0000 UTC m=+65.348711680 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-42drz" (UniqueName: "kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz") pod "network-check-target-dsznw" (UID: "3940f127-ec8e-40e7-85b4-1545fe589b92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 19:43:33.006620 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.006591 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:33.006620 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.006613 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:33.006840 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.006591 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:33.006840 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:33.006715 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:43:33.006907 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:33.006835 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsznw" podUID="3940f127-ec8e-40e7-85b4-1545fe589b92" Sep 05 19:43:33.006952 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:33.006901 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4l4q6" podUID="1ff9d244-8eb8-4b0d-ad2d-618e28803d5a" Sep 05 19:43:33.297923 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.297885 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeReady" Sep 05 19:43:33.298097 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.298052 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 05 19:43:33.346347 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.346315 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b7464dff9-hhskf"] Sep 05 19:43:33.376755 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.376724 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-rm5lj"] Sep 05 19:43:33.376942 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.376883 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.379674 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.379641 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rgzgk\"" Sep 05 19:43:33.379674 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.379662 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 05 19:43:33.379955 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.379770 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rzzf9"] Sep 05 19:43:33.379955 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.379844 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:43:33.380363 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.380343 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 05 19:43:33.380626 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.380607 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 05 19:43:33.386167 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.385964 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b7464dff9-hhskf"] Sep 05 19:43:33.386167 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.386042 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-rm5lj"] Sep 05 19:43:33.386384 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.386183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:33.386962 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.386458 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vkt5p"] Sep 05 19:43:33.388493 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.388471 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-64jqc\"" Sep 05 19:43:33.388589 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.388557 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 05 19:43:33.388763 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.388745 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 05 19:43:33.389357 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.389081 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 05 19:43:33.389357 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.389165 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6x9hn\"" Sep 05 19:43:33.392792 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.392773 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 05 19:43:33.394212 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.394191 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 05 19:43:33.403757 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.403730 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rzzf9"] Sep 05 19:43:33.403897 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.403777 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vkt5p"] Sep 05 19:43:33.403960 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.403906 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vkt5p" Sep 05 19:43:33.407958 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.407720 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 05 19:43:33.407958 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.407904 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 05 19:43:33.407958 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.407940 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7tzm\"" Sep 05 19:43:33.408188 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.408171 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 05 19:43:33.467143 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.467143 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467093 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:33.467143 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e4b881-669e-4df5-9feb-e59d740252bf-tmp-dir\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:33.467349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-certificates\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.467349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99e4b881-669e-4df5-9feb-e59d740252bf-config-volume\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:33.467349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467203 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ddc842b4-0f78-458c-ab9a-0f101d479c0b-installation-pull-secrets\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.467349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e1257f4-7b80-40cb-b11d-eb17481f369e-cert\") pod \"ingress-canary-vkt5p\" (UID: \"9e1257f4-7b80-40cb-b11d-eb17481f369e\") " pod="openshift-ingress-canary/ingress-canary-vkt5p" Sep 05 19:43:33.467349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gds8j\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-kube-api-access-gds8j\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.467349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:43:33.467349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjmwg\" (UniqueName: \"kubernetes.io/projected/9e1257f4-7b80-40cb-b11d-eb17481f369e-kube-api-access-xjmwg\") pod \"ingress-canary-vkt5p\" (UID: \"9e1257f4-7b80-40cb-b11d-eb17481f369e\") " pod="openshift-ingress-canary/ingress-canary-vkt5p" Sep 05 19:43:33.467349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ddc842b4-0f78-458c-ab9a-0f101d479c0b-trusted-ca\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.467349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467340 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ddc842b4-0f78-458c-ab9a-0f101d479c0b-image-registry-private-configuration\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.467604 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ddc842b4-0f78-458c-ab9a-0f101d479c0b-ca-trust-extracted\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.467604 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-bound-sa-token\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.467604 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f57d845b-1d2a-4c86-abf5-2f72a04b4532-nginx-conf\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:43:33.467604 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.467408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2dbp\" (UniqueName: \"kubernetes.io/projected/99e4b881-669e-4df5-9feb-e59d740252bf-kube-api-access-k2dbp\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:33.568702 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.568669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.568702 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.568712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:33.569026 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.568741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e4b881-669e-4df5-9feb-e59d740252bf-tmp-dir\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:33.569026 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:33.568874 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 19:43:33.569026 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:33.568963 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls podName:99e4b881-669e-4df5-9feb-e59d740252bf nodeName:}" failed. No retries permitted until 2026-09-05 19:43:34.068942403 +0000 UTC m=+34.554243258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls") pod "dns-default-rzzf9" (UID: "99e4b881-669e-4df5-9feb-e59d740252bf") : secret "dns-default-metrics-tls" not found Sep 05 19:43:33.569026 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:33.568984 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 19:43:33.569026 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:33.569003 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7464dff9-hhskf: secret "image-registry-tls" not found Sep 05 19:43:33.569260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569061 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e4b881-669e-4df5-9feb-e59d740252bf-tmp-dir\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:33.569260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.568882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-certificates\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.569260 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:33.569067 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls podName:ddc842b4-0f78-458c-ab9a-0f101d479c0b nodeName:}" failed. No retries permitted until 2026-09-05 19:43:34.069048776 +0000 UTC m=+34.554349655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls") pod "image-registry-7b7464dff9-hhskf" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b") : secret "image-registry-tls" not found Sep 05 19:43:33.569260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99e4b881-669e-4df5-9feb-e59d740252bf-config-volume\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:33.569260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569142 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ddc842b4-0f78-458c-ab9a-0f101d479c0b-installation-pull-secrets\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.569260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e1257f4-7b80-40cb-b11d-eb17481f369e-cert\") pod \"ingress-canary-vkt5p\" (UID: \"9e1257f4-7b80-40cb-b11d-eb17481f369e\") " pod="openshift-ingress-canary/ingress-canary-vkt5p" Sep 05 19:43:33.569260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gds8j\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-kube-api-access-gds8j\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.569260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:43:33.569260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjmwg\" (UniqueName: \"kubernetes.io/projected/9e1257f4-7b80-40cb-b11d-eb17481f369e-kube-api-access-xjmwg\") pod \"ingress-canary-vkt5p\" (UID: \"9e1257f4-7b80-40cb-b11d-eb17481f369e\") " pod="openshift-ingress-canary/ingress-canary-vkt5p" Sep 05 19:43:33.569260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ddc842b4-0f78-458c-ab9a-0f101d479c0b-trusted-ca\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.569683 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ddc842b4-0f78-458c-ab9a-0f101d479c0b-image-registry-private-configuration\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.569683 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ddc842b4-0f78-458c-ab9a-0f101d479c0b-ca-trust-extracted\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.569683 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-bound-sa-token\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.569683 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f57d845b-1d2a-4c86-abf5-2f72a04b4532-nginx-conf\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:43:33.569683 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2dbp\" (UniqueName: \"kubernetes.io/projected/99e4b881-669e-4df5-9feb-e59d740252bf-kube-api-access-k2dbp\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:33.569683 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569498 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-certificates\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.569976 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.569899 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99e4b881-669e-4df5-9feb-e59d740252bf-config-volume\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:33.570029 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:33.570013 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 05 19:43:33.570082 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:33.570077 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert podName:f57d845b-1d2a-4c86-abf5-2f72a04b4532 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:34.070061408 +0000 UTC m=+34.555362262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-rm5lj" (UID: "f57d845b-1d2a-4c86-abf5-2f72a04b4532") : secret "networking-console-plugin-cert" not found Sep 05 19:43:33.570357 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.570334 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ddc842b4-0f78-458c-ab9a-0f101d479c0b-ca-trust-extracted\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.570877 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.570850 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f57d845b-1d2a-4c86-abf5-2f72a04b4532-nginx-conf\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:43:33.571054 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.571033 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ddc842b4-0f78-458c-ab9a-0f101d479c0b-trusted-ca\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.573603 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.573579 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ddc842b4-0f78-458c-ab9a-0f101d479c0b-image-registry-private-configuration\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.573703 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.573647 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e1257f4-7b80-40cb-b11d-eb17481f369e-cert\") pod \"ingress-canary-vkt5p\" (UID: \"9e1257f4-7b80-40cb-b11d-eb17481f369e\") " pod="openshift-ingress-canary/ingress-canary-vkt5p" Sep 05 19:43:33.573703 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.573659 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ddc842b4-0f78-458c-ab9a-0f101d479c0b-installation-pull-secrets\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.581754 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.581727 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2dbp\" (UniqueName: \"kubernetes.io/projected/99e4b881-669e-4df5-9feb-e59d740252bf-kube-api-access-k2dbp\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:33.582094 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.582071 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-bound-sa-token\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.583474 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.583456 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjmwg\" (UniqueName: \"kubernetes.io/projected/9e1257f4-7b80-40cb-b11d-eb17481f369e-kube-api-access-xjmwg\") pod \"ingress-canary-vkt5p\" (UID: \"9e1257f4-7b80-40cb-b11d-eb17481f369e\") " pod="openshift-ingress-canary/ingress-canary-vkt5p" Sep 05 19:43:33.583640 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.583623 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gds8j\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-kube-api-access-gds8j\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:33.715994 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:33.715946 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vkt5p" Sep 05 19:43:34.030309 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:34.030275 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vkt5p"] Sep 05 19:43:34.034709 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:43:34.034681 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1257f4_7b80_40cb_b11d_eb17481f369e.slice/crio-6aeca1559b7abee7c0046a00824c3cbadf9d7360e37be7182a2269b28c433368 WatchSource:0}: Error finding container 6aeca1559b7abee7c0046a00824c3cbadf9d7360e37be7182a2269b28c433368: Status 404 returned error can't find the container with id 6aeca1559b7abee7c0046a00824c3cbadf9d7360e37be7182a2269b28c433368 Sep 05 19:43:34.073425 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:34.073385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:43:34.073626 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:34.073489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:34.073626 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:34.073518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:34.073626 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:34.073550 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 05 19:43:34.073626 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:34.073598 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 19:43:34.073626 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:34.073611 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 19:43:34.073626 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:34.073629 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert podName:f57d845b-1d2a-4c86-abf5-2f72a04b4532 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:35.0736083 +0000 UTC m=+35.558909155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-rm5lj" (UID: "f57d845b-1d2a-4c86-abf5-2f72a04b4532") : secret "networking-console-plugin-cert" not found Sep 05 19:43:34.073900 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:34.073630 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7464dff9-hhskf: secret "image-registry-tls" not found Sep 05 19:43:34.073900 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:34.073679 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls podName:ddc842b4-0f78-458c-ab9a-0f101d479c0b nodeName:}" failed. No retries permitted until 2026-09-05 19:43:35.073664728 +0000 UTC m=+35.558965584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls") pod "image-registry-7b7464dff9-hhskf" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b") : secret "image-registry-tls" not found Sep 05 19:43:34.073900 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:34.073743 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls podName:99e4b881-669e-4df5-9feb-e59d740252bf nodeName:}" failed. No retries permitted until 2026-09-05 19:43:35.073733619 +0000 UTC m=+35.559034473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls") pod "dns-default-rzzf9" (UID: "99e4b881-669e-4df5-9feb-e59d740252bf") : secret "dns-default-metrics-tls" not found Sep 05 19:43:34.268627 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:34.268592 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vkt5p" event={"ID":"9e1257f4-7b80-40cb-b11d-eb17481f369e","Type":"ContainerStarted","Data":"6aeca1559b7abee7c0046a00824c3cbadf9d7360e37be7182a2269b28c433368"} Sep 05 19:43:35.006503 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:35.006258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:43:35.007212 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:35.006305 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:43:35.007212 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:35.006349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:43:35.009483 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:35.009460 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gn74z\"" Sep 05 19:43:35.010834 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:35.010807 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 05 19:43:35.010949 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:35.010883 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 05 19:43:35.010949 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:35.010887 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-srspc\"" Sep 05 19:43:35.011062 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:35.010997 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 05 19:43:35.011062 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:35.011035 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 05 19:43:35.083374 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:35.083336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:35.083374 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:35.083376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:35.083920 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:35.083505 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 19:43:35.083920 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:35.083519 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 19:43:35.083920 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:35.083540 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7464dff9-hhskf: secret "image-registry-tls" not found Sep 05 19:43:35.083920 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:35.083563 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls podName:99e4b881-669e-4df5-9feb-e59d740252bf nodeName:}" failed. No retries permitted until 2026-09-05 19:43:37.083545985 +0000 UTC m=+37.568846841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls") pod "dns-default-rzzf9" (UID: "99e4b881-669e-4df5-9feb-e59d740252bf") : secret "dns-default-metrics-tls" not found Sep 05 19:43:35.083920 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:35.083580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:43:35.083920 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:35.083606 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls podName:ddc842b4-0f78-458c-ab9a-0f101d479c0b nodeName:}" failed. No retries permitted until 2026-09-05 19:43:37.083587405 +0000 UTC m=+37.568888256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls") pod "image-registry-7b7464dff9-hhskf" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b") : secret "image-registry-tls" not found Sep 05 19:43:35.083920 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:35.083643 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 05 19:43:35.083920 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:35.083674 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert podName:f57d845b-1d2a-4c86-abf5-2f72a04b4532 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:37.083666193 +0000 UTC m=+37.568967043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-rm5lj" (UID: "f57d845b-1d2a-4c86-abf5-2f72a04b4532") : secret "networking-console-plugin-cert" not found Sep 05 19:43:37.101574 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:37.101535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:37.101574 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:37.101579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:37.102072 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:37.101646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:43:37.102072 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:37.101718 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 19:43:37.102072 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:37.101742 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7464dff9-hhskf: secret "image-registry-tls" not found Sep 05 19:43:37.102072 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:37.101769 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 19:43:37.102072 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:37.101792 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 05 19:43:37.102072 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:37.101840 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls podName:ddc842b4-0f78-458c-ab9a-0f101d479c0b nodeName:}" failed. No retries permitted until 2026-09-05 19:43:41.101816508 +0000 UTC m=+41.587117360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls") pod "image-registry-7b7464dff9-hhskf" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b") : secret "image-registry-tls" not found Sep 05 19:43:37.102072 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:37.101861 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls podName:99e4b881-669e-4df5-9feb-e59d740252bf nodeName:}" failed. No retries permitted until 2026-09-05 19:43:41.101851414 +0000 UTC m=+41.587152266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls") pod "dns-default-rzzf9" (UID: "99e4b881-669e-4df5-9feb-e59d740252bf") : secret "dns-default-metrics-tls" not found Sep 05 19:43:37.102072 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:37.101879 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert podName:f57d845b-1d2a-4c86-abf5-2f72a04b4532 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:41.10186899 +0000 UTC m=+41.587169853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-rm5lj" (UID: "f57d845b-1d2a-4c86-abf5-2f72a04b4532") : secret "networking-console-plugin-cert" not found Sep 05 19:43:38.279038 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:38.279004 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vkt5p" event={"ID":"9e1257f4-7b80-40cb-b11d-eb17481f369e","Type":"ContainerStarted","Data":"0743cd0eb742cd1c9dcc5dd4f0c2f3fde29528a2f9ce364fd1c6dbbb8f80f861"} Sep 05 19:43:38.281766 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:38.281739 2578 generic.go:356] "Generic (PLEG): container finished" podID="51a28ad1-9a9b-47be-91b2-92dddc4a1408" containerID="5e1c457e9e89b7ab378e00f62f83a3fecb681f54fde957679f0acdb83b7ad100" exitCode=0 Sep 05 19:43:38.281905 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:38.281774 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpglw" event={"ID":"51a28ad1-9a9b-47be-91b2-92dddc4a1408","Type":"ContainerDied","Data":"5e1c457e9e89b7ab378e00f62f83a3fecb681f54fde957679f0acdb83b7ad100"} Sep 05 19:43:38.296214 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:38.296168 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vkt5p" podStartSLOduration=1.878718809 podStartE2EDuration="5.296154728s" totalImagesPullingTime="3.417435919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:38.295837854 +0000 UTC m=+38.781138727" watchObservedRunningTime="2026-09-05 19:43:38.296154728 +0000 UTC m=+38.781455601" Sep 05 19:43:39.286771 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:39.286733 2578 generic.go:356] "Generic (PLEG): container finished" podID="51a28ad1-9a9b-47be-91b2-92dddc4a1408" containerID="96253257c4f9d4bd69cdd1eb09abc59a71da7f100fc5731203c8d41224d26cae" exitCode=0 Sep 05 19:43:39.287443 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:39.286827 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpglw" event={"ID":"51a28ad1-9a9b-47be-91b2-92dddc4a1408","Type":"ContainerDied","Data":"96253257c4f9d4bd69cdd1eb09abc59a71da7f100fc5731203c8d41224d26cae"} Sep 05 19:43:40.292622 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:40.292578 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpglw" event={"ID":"51a28ad1-9a9b-47be-91b2-92dddc4a1408","Type":"ContainerStarted","Data":"9ca57bbf4634da0b2cce9632680c2a8ada23c2cbf51819aeab56c982d6348ecf"} Sep 05 19:43:40.321160 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:40.321102 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rpglw" podStartSLOduration=8.16138182 podStartE2EDuration="40.321084164s" totalImagesPullingTime="32.159702344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:40.31997819 +0000 UTC m=+40.805279088" watchObservedRunningTime="2026-09-05 19:43:40.321084164 +0000 UTC m=+40.806385034" Sep 05 19:43:41.132322 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:41.132259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:41.132322 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:41.132314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:41.132602 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:41.132362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:43:41.132602 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:41.132426 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 19:43:41.132602 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:41.132442 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 05 19:43:41.132602 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:41.132449 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7464dff9-hhskf: secret "image-registry-tls" not found Sep 05 19:43:41.132602 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:41.132491 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 19:43:41.132602 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:41.132498 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert podName:f57d845b-1d2a-4c86-abf5-2f72a04b4532 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:49.132484638 +0000 UTC m=+49.617785489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-rm5lj" (UID: "f57d845b-1d2a-4c86-abf5-2f72a04b4532") : secret "networking-console-plugin-cert" not found Sep 05 19:43:41.132602 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:41.132560 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls podName:ddc842b4-0f78-458c-ab9a-0f101d479c0b nodeName:}" failed. No retries permitted until 2026-09-05 19:43:49.132541599 +0000 UTC m=+49.617842459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls") pod "image-registry-7b7464dff9-hhskf" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b") : secret "image-registry-tls" not found Sep 05 19:43:41.132602 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:41.132576 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls podName:99e4b881-669e-4df5-9feb-e59d740252bf nodeName:}" failed. No retries permitted until 2026-09-05 19:43:49.13256657 +0000 UTC m=+49.617867427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls") pod "dns-default-rzzf9" (UID: "99e4b881-669e-4df5-9feb-e59d740252bf") : secret "dns-default-metrics-tls" not found Sep 05 19:43:49.193873 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:49.193829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:43:49.193873 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:49.193877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:43:49.194346 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:49.193942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:43:49.194346 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:49.193973 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 19:43:49.194346 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:49.193998 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7464dff9-hhskf: secret "image-registry-tls" not found Sep 05 19:43:49.194346 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:49.194030 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 19:43:49.194346 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:49.194063 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls podName:ddc842b4-0f78-458c-ab9a-0f101d479c0b nodeName:}" failed. No retries permitted until 2026-09-05 19:44:05.194047827 +0000 UTC m=+65.679348701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls") pod "image-registry-7b7464dff9-hhskf" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b") : secret "image-registry-tls" not found Sep 05 19:43:49.194346 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:49.194081 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 05 19:43:49.194346 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:49.194097 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls podName:99e4b881-669e-4df5-9feb-e59d740252bf nodeName:}" failed. No retries permitted until 2026-09-05 19:44:05.194083917 +0000 UTC m=+65.679384796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls") pod "dns-default-rzzf9" (UID: "99e4b881-669e-4df5-9feb-e59d740252bf") : secret "dns-default-metrics-tls" not found Sep 05 19:43:49.194346 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:43:49.194139 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert podName:f57d845b-1d2a-4c86-abf5-2f72a04b4532 nodeName:}" failed. No retries permitted until 2026-09-05 19:44:05.194125551 +0000 UTC m=+65.679426416 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-rm5lj" (UID: "f57d845b-1d2a-4c86-abf5-2f72a04b4532") : secret "networking-console-plugin-cert" not found Sep 05 19:43:59.263360 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:43:59.263333 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn472" Sep 05 19:44:04.811878 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:04.811815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:44:04.812361 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:04.811925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:44:04.815214 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:04.815185 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 05 19:44:04.815338 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:04.815241 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 05 19:44:04.822640 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:04.822611 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 05 19:44:04.822741 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:04.822676 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs podName:fe2aaad4-dc65-4f15-be9a-9fe0babba5b7 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:08.822659479 +0000 UTC m=+129.307960343 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs") pod "network-metrics-daemon-89gvd" (UID: "fe2aaad4-dc65-4f15-be9a-9fe0babba5b7") : secret "metrics-daemon-secret" not found Sep 05 19:44:04.825609 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:04.825586 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ff9d244-8eb8-4b0d-ad2d-618e28803d5a-original-pull-secret\") pod \"global-pull-secret-syncer-4l4q6\" (UID: \"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a\") " pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:44:04.912619 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:04.912577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42drz\" (UniqueName: \"kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz\") pod \"network-check-target-dsznw\" (UID: \"3940f127-ec8e-40e7-85b4-1545fe589b92\") " pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:44:04.916195 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:04.916176 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 05 19:44:04.926074 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:04.926052 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 05 19:44:04.936498 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:04.936469 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42drz\" (UniqueName: \"kubernetes.io/projected/3940f127-ec8e-40e7-85b4-1545fe589b92-kube-api-access-42drz\") pod \"network-check-target-dsznw\" (UID: \"3940f127-ec8e-40e7-85b4-1545fe589b92\") " pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:44:05.023035 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:05.023003 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gn74z\"" Sep 05 19:44:05.028469 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:05.028451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4l4q6" Sep 05 19:44:05.030593 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:05.030572 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:44:05.214745 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:05.214705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:44:05.214745 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:05.214742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:44:05.215021 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:05.214781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:44:05.215021 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:05.214884 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 19:44:05.215021 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:05.214896 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 05 19:44:05.215021 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:05.214906 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7464dff9-hhskf: secret "image-registry-tls" not found Sep 05 19:44:05.215021 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:05.214909 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 19:44:05.215021 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:05.214959 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert podName:f57d845b-1d2a-4c86-abf5-2f72a04b4532 nodeName:}" failed. No retries permitted until 2026-09-05 19:44:37.214939716 +0000 UTC m=+97.700240566 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-rm5lj" (UID: "f57d845b-1d2a-4c86-abf5-2f72a04b4532") : secret "networking-console-plugin-cert" not found Sep 05 19:44:05.215021 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:05.214972 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls podName:99e4b881-669e-4df5-9feb-e59d740252bf nodeName:}" failed. No retries permitted until 2026-09-05 19:44:37.214966198 +0000 UTC m=+97.700267048 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls") pod "dns-default-rzzf9" (UID: "99e4b881-669e-4df5-9feb-e59d740252bf") : secret "dns-default-metrics-tls" not found Sep 05 19:44:05.215021 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:05.214981 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls podName:ddc842b4-0f78-458c-ab9a-0f101d479c0b nodeName:}" failed. No retries permitted until 2026-09-05 19:44:37.214976092 +0000 UTC m=+97.700276943 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls") pod "image-registry-7b7464dff9-hhskf" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b") : secret "image-registry-tls" not found Sep 05 19:44:05.308632 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:05.308599 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4l4q6"] Sep 05 19:44:05.311429 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:05.311405 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dsznw"] Sep 05 19:44:05.312174 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:44:05.312085 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff9d244_8eb8_4b0d_ad2d_618e28803d5a.slice/crio-32861d48c8260208fff9c7090083e208b1b783f2a167685616c7f8a8f23492b1 WatchSource:0}: Error finding container 32861d48c8260208fff9c7090083e208b1b783f2a167685616c7f8a8f23492b1: Status 404 returned error can't find the container with id 32861d48c8260208fff9c7090083e208b1b783f2a167685616c7f8a8f23492b1 Sep 05 19:44:05.314353 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:44:05.314301 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3940f127_ec8e_40e7_85b4_1545fe589b92.slice/crio-175acf7545fce08508c2b6cc69a17f34d03eff3e1e9031c92e47669d1629a31a WatchSource:0}: Error finding container 175acf7545fce08508c2b6cc69a17f34d03eff3e1e9031c92e47669d1629a31a: Status 404 returned error can't find the container with id 175acf7545fce08508c2b6cc69a17f34d03eff3e1e9031c92e47669d1629a31a Sep 05 19:44:05.355359 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:05.355266 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dsznw" event={"ID":"3940f127-ec8e-40e7-85b4-1545fe589b92","Type":"ContainerStarted","Data":"175acf7545fce08508c2b6cc69a17f34d03eff3e1e9031c92e47669d1629a31a"} Sep 05 19:44:05.356318 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:05.356287 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4l4q6" event={"ID":"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a","Type":"ContainerStarted","Data":"32861d48c8260208fff9c7090083e208b1b783f2a167685616c7f8a8f23492b1"} Sep 05 19:44:10.371871 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:10.371768 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dsznw" event={"ID":"3940f127-ec8e-40e7-85b4-1545fe589b92","Type":"ContainerStarted","Data":"4100fde8bbdc40684d64a4786099eb41c86b805db41b76e7546a4a81204d8ccb"} Sep 05 19:44:10.372364 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:10.371885 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:44:10.373213 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:10.373181 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4l4q6" event={"ID":"1ff9d244-8eb8-4b0d-ad2d-618e28803d5a","Type":"ContainerStarted","Data":"a69c87d2777865220bdc1617f379449f3ee4d92357bfa3a13f359ac96d11f6c2"} Sep 05 19:44:10.395267 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:10.395211 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dsznw" podStartSLOduration=65.594862567 podStartE2EDuration="1m10.395195407s" totalImagesPullingTime="4.80033284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:44:10.392991302 +0000 UTC m=+70.878292176" watchObservedRunningTime="2026-09-05 19:44:10.395195407 +0000 UTC m=+70.880496280" Sep 05 19:44:10.421970 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:10.421913 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4l4q6" podStartSLOduration=65.608166912 podStartE2EDuration="1m10.421896829s" totalImagesPullingTime="4.813729917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:44:10.420775638 +0000 UTC m=+70.906076513" watchObservedRunningTime="2026-09-05 19:44:10.421896829 +0000 UTC m=+70.907197696" Sep 05 19:44:37.247432 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:37.247345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:44:37.247432 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:37.247386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:44:37.247933 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:37.247499 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:44:37.247933 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:37.247552 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 19:44:37.247933 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:37.247581 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7464dff9-hhskf: secret "image-registry-tls" not found Sep 05 19:44:37.247933 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:37.247585 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 19:44:37.247933 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:37.247594 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 05 19:44:37.247933 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:37.247650 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls podName:ddc842b4-0f78-458c-ab9a-0f101d479c0b nodeName:}" failed. No retries permitted until 2026-09-05 19:45:41.247627488 +0000 UTC m=+161.732928355 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls") pod "image-registry-7b7464dff9-hhskf" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b") : secret "image-registry-tls" not found Sep 05 19:44:37.247933 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:37.247669 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls podName:99e4b881-669e-4df5-9feb-e59d740252bf nodeName:}" failed. No retries permitted until 2026-09-05 19:45:41.24765961 +0000 UTC m=+161.732960464 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls") pod "dns-default-rzzf9" (UID: "99e4b881-669e-4df5-9feb-e59d740252bf") : secret "dns-default-metrics-tls" not found Sep 05 19:44:37.247933 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:44:37.247685 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert podName:f57d845b-1d2a-4c86-abf5-2f72a04b4532 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:41.24767541 +0000 UTC m=+161.732976267 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-rm5lj" (UID: "f57d845b-1d2a-4c86-abf5-2f72a04b4532") : secret "networking-console-plugin-cert" not found Sep 05 19:44:41.378712 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:44:41.378681 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dsznw" Sep 05 19:45:08.877946 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:08.877902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:45:08.878411 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:08.878047 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 05 19:45:08.878411 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:08.878110 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs podName:fe2aaad4-dc65-4f15-be9a-9fe0babba5b7 nodeName:}" failed. No retries permitted until 2026-09-05 19:47:10.878094403 +0000 UTC m=+251.363395254 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs") pod "network-metrics-daemon-89gvd" (UID: "fe2aaad4-dc65-4f15-be9a-9fe0babba5b7") : secret "metrics-daemon-secret" not found Sep 05 19:45:35.636621 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.636583 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-nptkb"] Sep 05 19:45:35.639294 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.639277 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.643505 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.643483 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 05 19:45:35.643849 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.643833 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t7kh4\"" Sep 05 19:45:35.644303 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.644287 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 05 19:45:35.645417 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.644863 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 05 19:45:35.645417 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.644863 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 05 19:45:35.649305 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.649271 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 05 19:45:35.654981 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.654959 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-nptkb"] Sep 05 19:45:35.729025 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.728990 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7"] Sep 05 19:45:35.731958 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.731940 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:35.734569 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.734540 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 05 19:45:35.734708 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.734545 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 05 19:45:35.734708 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.734546 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 05 19:45:35.734708 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.734548 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zjhbp\"" Sep 05 19:45:35.739403 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.739382 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7"] Sep 05 19:45:35.775820 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.775774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbbmr\" (UniqueName: \"kubernetes.io/projected/66922fa5-4d55-479f-b0b5-baef39b284f3-kube-api-access-wbbmr\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.775970 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.775891 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/66922fa5-4d55-479f-b0b5-baef39b284f3-snapshots\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.775970 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.775928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66922fa5-4d55-479f-b0b5-baef39b284f3-tmp\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.775970 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.775949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66922fa5-4d55-479f-b0b5-baef39b284f3-serving-cert\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.776117 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.775993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66922fa5-4d55-479f-b0b5-baef39b284f3-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.776117 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.776042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66922fa5-4d55-479f-b0b5-baef39b284f3-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.877317 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.877269 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66922fa5-4d55-479f-b0b5-baef39b284f3-tmp\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.877317 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.877320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66922fa5-4d55-479f-b0b5-baef39b284f3-serving-cert\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.877469 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.877366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66922fa5-4d55-479f-b0b5-baef39b284f3-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.877469 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.877387 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6ptf7\" (UID: \"c61629c9-f748-44bb-abfe-6d1c047d5f02\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:35.877469 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.877407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm46n\" (UniqueName: \"kubernetes.io/projected/c61629c9-f748-44bb-abfe-6d1c047d5f02-kube-api-access-pm46n\") pod \"cluster-samples-operator-7cdcfcf76c-6ptf7\" (UID: \"c61629c9-f748-44bb-abfe-6d1c047d5f02\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:35.877469 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.877436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66922fa5-4d55-479f-b0b5-baef39b284f3-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.877594 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.877525 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbbmr\" (UniqueName: \"kubernetes.io/projected/66922fa5-4d55-479f-b0b5-baef39b284f3-kube-api-access-wbbmr\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.877638 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.877619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/66922fa5-4d55-479f-b0b5-baef39b284f3-snapshots\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.877742 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.877722 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66922fa5-4d55-479f-b0b5-baef39b284f3-tmp\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.878088 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.878063 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66922fa5-4d55-479f-b0b5-baef39b284f3-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.878176 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.878100 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/66922fa5-4d55-479f-b0b5-baef39b284f3-snapshots\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.878270 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.878254 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66922fa5-4d55-479f-b0b5-baef39b284f3-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.879980 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.879955 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66922fa5-4d55-479f-b0b5-baef39b284f3-serving-cert\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.885741 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.885716 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbbmr\" (UniqueName: \"kubernetes.io/projected/66922fa5-4d55-479f-b0b5-baef39b284f3-kube-api-access-wbbmr\") pod \"insights-operator-76bcd6ccf5-nptkb\" (UID: \"66922fa5-4d55-479f-b0b5-baef39b284f3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.947828 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.947718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" Sep 05 19:45:35.978155 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.978120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6ptf7\" (UID: \"c61629c9-f748-44bb-abfe-6d1c047d5f02\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:35.978155 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.978156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm46n\" (UniqueName: \"kubernetes.io/projected/c61629c9-f748-44bb-abfe-6d1c047d5f02-kube-api-access-pm46n\") pod \"cluster-samples-operator-7cdcfcf76c-6ptf7\" (UID: \"c61629c9-f748-44bb-abfe-6d1c047d5f02\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:35.978358 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:35.978286 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 05 19:45:35.978394 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:35.978365 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls podName:c61629c9-f748-44bb-abfe-6d1c047d5f02 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:36.478346232 +0000 UTC m=+156.963647103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6ptf7" (UID: "c61629c9-f748-44bb-abfe-6d1c047d5f02") : secret "samples-operator-tls" not found Sep 05 19:45:35.989417 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:35.989382 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm46n\" (UniqueName: \"kubernetes.io/projected/c61629c9-f748-44bb-abfe-6d1c047d5f02-kube-api-access-pm46n\") pod \"cluster-samples-operator-7cdcfcf76c-6ptf7\" (UID: \"c61629c9-f748-44bb-abfe-6d1c047d5f02\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:36.217821 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:36.217726 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-nptkb"] Sep 05 19:45:36.220555 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:45:36.220531 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66922fa5_4d55_479f_b0b5_baef39b284f3.slice/crio-fb66cde2b62d9da3343a0bbbed4d67e9fb27888f82763b91be29856c4240d918 WatchSource:0}: Error finding container fb66cde2b62d9da3343a0bbbed4d67e9fb27888f82763b91be29856c4240d918: Status 404 returned error can't find the container with id fb66cde2b62d9da3343a0bbbed4d67e9fb27888f82763b91be29856c4240d918 Sep 05 19:45:36.393713 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:36.393660 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" podUID="ddc842b4-0f78-458c-ab9a-0f101d479c0b" Sep 05 19:45:36.401834 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:36.401783 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" podUID="f57d845b-1d2a-4c86-abf5-2f72a04b4532" Sep 05 19:45:36.408984 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:36.408957 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rzzf9" podUID="99e4b881-669e-4df5-9feb-e59d740252bf" Sep 05 19:45:36.482633 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:36.482544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6ptf7\" (UID: \"c61629c9-f748-44bb-abfe-6d1c047d5f02\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:36.482840 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:36.482701 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 05 19:45:36.482840 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:36.482767 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls podName:c61629c9-f748-44bb-abfe-6d1c047d5f02 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:37.482752889 +0000 UTC m=+157.968053745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6ptf7" (UID: "c61629c9-f748-44bb-abfe-6d1c047d5f02") : secret "samples-operator-tls" not found Sep 05 19:45:36.581502 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:36.581471 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:45:36.581502 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:36.581502 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:45:36.581697 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:36.581492 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" event={"ID":"66922fa5-4d55-479f-b0b5-baef39b284f3","Type":"ContainerStarted","Data":"fb66cde2b62d9da3343a0bbbed4d67e9fb27888f82763b91be29856c4240d918"} Sep 05 19:45:36.581697 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:36.581480 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rzzf9" Sep 05 19:45:37.489728 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:37.489690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6ptf7\" (UID: \"c61629c9-f748-44bb-abfe-6d1c047d5f02\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:37.490176 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:37.489844 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 05 19:45:37.490176 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:37.489908 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls podName:c61629c9-f748-44bb-abfe-6d1c047d5f02 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:39.489890475 +0000 UTC m=+159.975191329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6ptf7" (UID: "c61629c9-f748-44bb-abfe-6d1c047d5f02") : secret "samples-operator-tls" not found Sep 05 19:45:38.034566 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:38.034422 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-89gvd" podUID="fe2aaad4-dc65-4f15-be9a-9fe0babba5b7" Sep 05 19:45:38.588165 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:38.588120 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" event={"ID":"66922fa5-4d55-479f-b0b5-baef39b284f3","Type":"ContainerStarted","Data":"ff98802480a7b7f0ed2f769e8b450c0d3419c7962366e12c5886d1cd0e4e3f3d"} Sep 05 19:45:38.607656 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:38.607598 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" podStartSLOduration=1.644159851 podStartE2EDuration="3.607580848s" totalImagesPullingTime="1.963420997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:45:38.606769522 +0000 UTC m=+159.092070397" watchObservedRunningTime="2026-09-05 19:45:38.607580848 +0000 UTC m=+159.092881720" Sep 05 19:45:39.503108 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.503063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6ptf7\" (UID: \"c61629c9-f748-44bb-abfe-6d1c047d5f02\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:39.503327 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:39.503195 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 05 19:45:39.503327 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:39.503277 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls podName:c61629c9-f748-44bb-abfe-6d1c047d5f02 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:43.503258871 +0000 UTC m=+163.988559733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6ptf7" (UID: "c61629c9-f748-44bb-abfe-6d1c047d5f02") : secret "samples-operator-tls" not found Sep 05 19:45:39.565593 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.565556 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-85hn8"] Sep 05 19:45:39.568888 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.568867 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.571712 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.571680 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 05 19:45:39.571986 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.571967 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 05 19:45:39.571986 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.571978 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wszhj\"" Sep 05 19:45:39.580686 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.580661 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-85hn8"] Sep 05 19:45:39.603633 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.603587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.604148 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.603692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.604148 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.603717 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7n4g\" (UniqueName: \"kubernetes.io/projected/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-api-access-x7n4g\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.604148 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.603829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9891d97-10d1-45d7-92c2-b2e28465c386-crio-socket\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.604148 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.603874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9891d97-10d1-45d7-92c2-b2e28465c386-data-volume\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.704339 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.704302 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9891d97-10d1-45d7-92c2-b2e28465c386-crio-socket\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.704339 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.704340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9891d97-10d1-45d7-92c2-b2e28465c386-data-volume\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.704542 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.704380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.704542 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.704399 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9891d97-10d1-45d7-92c2-b2e28465c386-crio-socket\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.704542 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.704428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.704542 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.704465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7n4g\" (UniqueName: \"kubernetes.io/projected/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-api-access-x7n4g\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.704542 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:39.704508 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 19:45:39.704542 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:39.704532 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 05 19:45:39.704746 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:39.704553 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls podName:a9891d97-10d1-45d7-92c2-b2e28465c386 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:40.20453955 +0000 UTC m=+160.689840419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85hn8" (UID: "a9891d97-10d1-45d7-92c2-b2e28465c386") : secret "insights-runtime-extractor-tls" not found Sep 05 19:45:39.704746 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:39.704604 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm podName:a9891d97-10d1-45d7-92c2-b2e28465c386 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:40.204589396 +0000 UTC m=+160.689890247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85hn8" (UID: "a9891d97-10d1-45d7-92c2-b2e28465c386") : configmap "kube-rbac-proxy" not found Sep 05 19:45:39.704746 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.704728 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9891d97-10d1-45d7-92c2-b2e28465c386-data-volume\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:39.715710 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:39.715684 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7n4g\" (UniqueName: \"kubernetes.io/projected/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-api-access-x7n4g\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:40.207454 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:40.207415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:40.207647 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:40.207540 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 05 19:45:40.207647 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:40.207575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:40.207647 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:40.207607 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm podName:a9891d97-10d1-45d7-92c2-b2e28465c386 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:41.207591047 +0000 UTC m=+161.692891931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85hn8" (UID: "a9891d97-10d1-45d7-92c2-b2e28465c386") : configmap "kube-rbac-proxy" not found Sep 05 19:45:40.207771 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:40.207666 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 19:45:40.207771 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:40.207708 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls podName:a9891d97-10d1-45d7-92c2-b2e28465c386 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:41.207697286 +0000 UTC m=+161.692998137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85hn8" (UID: "a9891d97-10d1-45d7-92c2-b2e28465c386") : secret "insights-runtime-extractor-tls" not found Sep 05 19:45:41.216244 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:41.216199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:41.216730 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:41.216267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:41.216730 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:41.216339 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 05 19:45:41.216730 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:41.216391 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 19:45:41.216730 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:41.216408 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm podName:a9891d97-10d1-45d7-92c2-b2e28465c386 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:43.216393015 +0000 UTC m=+163.701693887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85hn8" (UID: "a9891d97-10d1-45d7-92c2-b2e28465c386") : configmap "kube-rbac-proxy" not found Sep 05 19:45:41.216730 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:41.216429 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls podName:a9891d97-10d1-45d7-92c2-b2e28465c386 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:43.216418357 +0000 UTC m=+163.701719207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85hn8" (UID: "a9891d97-10d1-45d7-92c2-b2e28465c386") : secret "insights-runtime-extractor-tls" not found Sep 05 19:45:41.317538 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:41.317500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:45:41.317719 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:41.317567 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls\") pod \"image-registry-7b7464dff9-hhskf\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:45:41.317719 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:41.317592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:45:41.317719 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:41.317657 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 05 19:45:41.317719 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:41.317697 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 19:45:41.317897 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:41.317658 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 19:45:41.317897 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:41.317749 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b7464dff9-hhskf: secret "image-registry-tls" not found Sep 05 19:45:41.317897 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:41.317737 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert podName:f57d845b-1d2a-4c86-abf5-2f72a04b4532 nodeName:}" failed. No retries permitted until 2026-09-05 19:47:43.317718474 +0000 UTC m=+283.803019346 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-rm5lj" (UID: "f57d845b-1d2a-4c86-abf5-2f72a04b4532") : secret "networking-console-plugin-cert" not found Sep 05 19:45:41.317897 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:41.317786 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls podName:99e4b881-669e-4df5-9feb-e59d740252bf nodeName:}" failed. No retries permitted until 2026-09-05 19:47:43.317776749 +0000 UTC m=+283.803077607 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls") pod "dns-default-rzzf9" (UID: "99e4b881-669e-4df5-9feb-e59d740252bf") : secret "dns-default-metrics-tls" not found Sep 05 19:45:41.317897 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:41.317820 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls podName:ddc842b4-0f78-458c-ab9a-0f101d479c0b nodeName:}" failed. No retries permitted until 2026-09-05 19:47:43.317792025 +0000 UTC m=+283.803092881 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls") pod "image-registry-7b7464dff9-hhskf" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b") : secret "image-registry-tls" not found Sep 05 19:45:41.337858 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:41.337829 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-26ptb_593dff6f-9c82-4da6-90cb-35032860e715/dns-node-resolver/0.log" Sep 05 19:45:42.537505 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:42.537476 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qc2c8_b04d888b-736a-42e7-8a96-81d738b05abb/node-ca/0.log" Sep 05 19:45:43.235285 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:43.235234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:43.235474 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:43.235387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:43.235474 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:43.235394 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 19:45:43.235474 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:43.235458 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls podName:a9891d97-10d1-45d7-92c2-b2e28465c386 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:47.235441661 +0000 UTC m=+167.720742511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85hn8" (UID: "a9891d97-10d1-45d7-92c2-b2e28465c386") : secret "insights-runtime-extractor-tls" not found Sep 05 19:45:43.235582 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:43.235474 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 05 19:45:43.235582 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:43.235516 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm podName:a9891d97-10d1-45d7-92c2-b2e28465c386 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:47.235505011 +0000 UTC m=+167.720805862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85hn8" (UID: "a9891d97-10d1-45d7-92c2-b2e28465c386") : configmap "kube-rbac-proxy" not found Sep 05 19:45:43.536818 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:43.536771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6ptf7\" (UID: \"c61629c9-f748-44bb-abfe-6d1c047d5f02\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:43.537020 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:43.536927 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 05 19:45:43.537020 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:43.536995 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls podName:c61629c9-f748-44bb-abfe-6d1c047d5f02 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:51.53697866 +0000 UTC m=+172.022279520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6ptf7" (UID: "c61629c9-f748-44bb-abfe-6d1c047d5f02") : secret "samples-operator-tls" not found Sep 05 19:45:43.537195 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:43.537177 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vkt5p_9e1257f4-7b80-40cb-b11d-eb17481f369e/serve-healthcheck-canary/0.log" Sep 05 19:45:47.268658 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:47.268619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:47.269074 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:47.268716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:47.269074 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:47.268736 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 05 19:45:47.269074 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:47.268826 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm podName:a9891d97-10d1-45d7-92c2-b2e28465c386 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:55.268792779 +0000 UTC m=+175.754093635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85hn8" (UID: "a9891d97-10d1-45d7-92c2-b2e28465c386") : configmap "kube-rbac-proxy" not found Sep 05 19:45:47.269074 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:47.268852 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 19:45:47.269074 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:47.268900 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls podName:a9891d97-10d1-45d7-92c2-b2e28465c386 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:55.268884382 +0000 UTC m=+175.754185255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85hn8" (UID: "a9891d97-10d1-45d7-92c2-b2e28465c386") : secret "insights-runtime-extractor-tls" not found Sep 05 19:45:51.611140 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:51.611097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6ptf7\" (UID: \"c61629c9-f748-44bb-abfe-6d1c047d5f02\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:51.613477 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:51.613444 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c61629c9-f748-44bb-abfe-6d1c047d5f02-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6ptf7\" (UID: \"c61629c9-f748-44bb-abfe-6d1c047d5f02\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:51.640923 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:51.640888 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" Sep 05 19:45:51.914668 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:51.914592 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7"] Sep 05 19:45:52.624492 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:52.624448 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" event={"ID":"c61629c9-f748-44bb-abfe-6d1c047d5f02","Type":"ContainerStarted","Data":"a2a9578b8bdf9613b9cd8e14329cad68206179832bafde19c75292d03bff7d54"} Sep 05 19:45:53.006572 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:53.006522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:45:53.629440 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:53.629396 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" event={"ID":"c61629c9-f748-44bb-abfe-6d1c047d5f02","Type":"ContainerStarted","Data":"75e5bf28d8a816fd0dae7394dfb378f5e7aa99377b4119467767ab63a2793907"} Sep 05 19:45:54.634018 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:54.633978 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" event={"ID":"c61629c9-f748-44bb-abfe-6d1c047d5f02","Type":"ContainerStarted","Data":"bccd8f83c302bbfa641e2b17a5a04fa00f5ab5fc66670247ff1d950ce2a0b1b5"} Sep 05 19:45:54.652602 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:54.652545 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6ptf7" podStartSLOduration=18.069720516 podStartE2EDuration="19.65253109s" totalImagesPullingTime="1.582810574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:45:54.652360588 +0000 UTC m=+175.137661462" watchObservedRunningTime="2026-09-05 19:45:54.65253109 +0000 UTC m=+175.137831959" Sep 05 19:45:55.343738 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:55.343692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:55.343948 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:55.343834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:45:55.343998 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:55.343956 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 19:45:55.344037 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:45:55.344020 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls podName:a9891d97-10d1-45d7-92c2-b2e28465c386 nodeName:}" failed. No retries permitted until 2026-09-05 19:46:11.344003985 +0000 UTC m=+191.829304837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85hn8" (UID: "a9891d97-10d1-45d7-92c2-b2e28465c386") : secret "insights-runtime-extractor-tls" not found Sep 05 19:45:55.344217 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:45:55.344197 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9891d97-10d1-45d7-92c2-b2e28465c386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:46:04.666381 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:04.666349 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-n5qrf"] Sep 05 19:46:04.669489 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:04.669473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-n5qrf" Sep 05 19:46:04.673025 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:04.672999 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 05 19:46:04.673141 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:04.673033 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 05 19:46:04.673478 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:04.673460 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-tmqw8\"" Sep 05 19:46:04.688975 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:04.688921 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-n5qrf"] Sep 05 19:46:04.821551 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:04.821510 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x566\" (UniqueName: \"kubernetes.io/projected/7ae64473-c8b8-42cf-bc67-c7d67f744d2c-kube-api-access-4x566\") pod \"downloads-85d98c998f-n5qrf\" (UID: \"7ae64473-c8b8-42cf-bc67-c7d67f744d2c\") " pod="openshift-console/downloads-85d98c998f-n5qrf" Sep 05 19:46:04.922953 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:04.922860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x566\" (UniqueName: \"kubernetes.io/projected/7ae64473-c8b8-42cf-bc67-c7d67f744d2c-kube-api-access-4x566\") pod \"downloads-85d98c998f-n5qrf\" (UID: \"7ae64473-c8b8-42cf-bc67-c7d67f744d2c\") " pod="openshift-console/downloads-85d98c998f-n5qrf" Sep 05 19:46:04.931712 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:04.931678 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x566\" (UniqueName: \"kubernetes.io/projected/7ae64473-c8b8-42cf-bc67-c7d67f744d2c-kube-api-access-4x566\") pod \"downloads-85d98c998f-n5qrf\" (UID: \"7ae64473-c8b8-42cf-bc67-c7d67f744d2c\") " pod="openshift-console/downloads-85d98c998f-n5qrf" Sep 05 19:46:04.978273 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:04.978219 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-n5qrf" Sep 05 19:46:05.250446 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:05.250360 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-n5qrf"] Sep 05 19:46:05.254281 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:46:05.254251 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae64473_c8b8_42cf_bc67_c7d67f744d2c.slice/crio-ba0d0fe65333ccdad05933bc43e0c8be1d75d81e0d96462ff4bc8254ab48deda WatchSource:0}: Error finding container ba0d0fe65333ccdad05933bc43e0c8be1d75d81e0d96462ff4bc8254ab48deda: Status 404 returned error can't find the container with id ba0d0fe65333ccdad05933bc43e0c8be1d75d81e0d96462ff4bc8254ab48deda Sep 05 19:46:05.668098 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:05.668058 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-n5qrf" event={"ID":"7ae64473-c8b8-42cf-bc67-c7d67f744d2c","Type":"ContainerStarted","Data":"ba0d0fe65333ccdad05933bc43e0c8be1d75d81e0d96462ff4bc8254ab48deda"} Sep 05 19:46:09.849263 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:09.849225 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc"] Sep 05 19:46:09.852751 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:09.852727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc" Sep 05 19:46:09.855776 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:09.855752 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zlmgp\"" Sep 05 19:46:09.855988 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:09.855968 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 05 19:46:09.863922 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:09.863901 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc"] Sep 05 19:46:09.964288 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:09.964247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-fqlsc\" (UID: \"1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc" Sep 05 19:46:10.065110 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:10.065070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-fqlsc\" (UID: \"1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc" Sep 05 19:46:10.065299 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:46:10.065209 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 05 19:46:10.065299 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:46:10.065280 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9-tls-certificates podName:1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9 nodeName:}" failed. No retries permitted until 2026-09-05 19:46:10.56526138 +0000 UTC m=+191.050562236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-fqlsc" (UID: "1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9") : secret "prometheus-operator-admission-webhook-tls" not found Sep 05 19:46:10.569502 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:10.569453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-fqlsc\" (UID: \"1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc" Sep 05 19:46:10.572203 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:10.572167 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-fqlsc\" (UID: \"1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc" Sep 05 19:46:10.763901 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:10.763861 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc" Sep 05 19:46:11.046132 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:11.046042 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc"] Sep 05 19:46:11.049121 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:46:11.049079 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf40c05_15d3_46fa_9aeb_5296fdd7b3c9.slice/crio-018c9017666c571ba8a19dee12f1bbdbe78c798f21a9518ff00da0a87691690a WatchSource:0}: Error finding container 018c9017666c571ba8a19dee12f1bbdbe78c798f21a9518ff00da0a87691690a: Status 404 returned error can't find the container with id 018c9017666c571ba8a19dee12f1bbdbe78c798f21a9518ff00da0a87691690a Sep 05 19:46:11.376957 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:11.376907 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:46:11.379473 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:11.379440 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9891d97-10d1-45d7-92c2-b2e28465c386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85hn8\" (UID: \"a9891d97-10d1-45d7-92c2-b2e28465c386\") " pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:46:11.681395 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:11.681313 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wszhj\"" Sep 05 19:46:11.688621 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:11.688584 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-85hn8" Sep 05 19:46:11.690432 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:11.690395 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc" event={"ID":"1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9","Type":"ContainerStarted","Data":"018c9017666c571ba8a19dee12f1bbdbe78c798f21a9518ff00da0a87691690a"} Sep 05 19:46:11.979065 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:11.978984 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-85hn8"] Sep 05 19:46:12.193675 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:46:12.193641 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9891d97_10d1_45d7_92c2_b2e28465c386.slice/crio-b3415089414bf20aede7d384517c9516a8b935a576c472edef69b8814500928a WatchSource:0}: Error finding container b3415089414bf20aede7d384517c9516a8b935a576c472edef69b8814500928a: Status 404 returned error can't find the container with id b3415089414bf20aede7d384517c9516a8b935a576c472edef69b8814500928a Sep 05 19:46:12.642609 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.642566 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68cc7f86db-2vvsf"] Sep 05 19:46:12.645811 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.645764 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.648651 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.648629 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 05 19:46:12.650266 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.650070 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 05 19:46:12.650266 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.650082 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wf7bw\"" Sep 05 19:46:12.650266 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.650082 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 05 19:46:12.650266 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.650142 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 05 19:46:12.650266 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.650157 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 05 19:46:12.654967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.654937 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68cc7f86db-2vvsf"] Sep 05 19:46:12.696110 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.696036 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc" event={"ID":"1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9","Type":"ContainerStarted","Data":"c63ae4cbad48611b9207498420560c7025a5d8fd7091a3962b65e911f3808d7f"} Sep 05 19:46:12.696291 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.696193 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc" Sep 05 19:46:12.697925 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.697896 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-85hn8" event={"ID":"a9891d97-10d1-45d7-92c2-b2e28465c386","Type":"ContainerStarted","Data":"821cb1ff727372244bd38057b41f474284a6f10966a8a907d5d0d13296b2cfeb"} Sep 05 19:46:12.698050 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.697954 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-85hn8" event={"ID":"a9891d97-10d1-45d7-92c2-b2e28465c386","Type":"ContainerStarted","Data":"b3415089414bf20aede7d384517c9516a8b935a576c472edef69b8814500928a"} Sep 05 19:46:12.702024 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.702001 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc" Sep 05 19:46:12.712522 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.712472 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fqlsc" podStartSLOduration=2.5234856150000002 podStartE2EDuration="3.712458949s" totalImagesPullingTime="1.188973334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:12.711685461 +0000 UTC m=+193.196986334" watchObservedRunningTime="2026-09-05 19:46:12.712458949 +0000 UTC m=+193.197759835" Sep 05 19:46:12.788292 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.788234 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhlg\" (UniqueName: \"kubernetes.io/projected/3de746a5-fc32-4c55-97d4-cb55fe39360b-kube-api-access-5hhlg\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.788292 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.788279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-serving-cert\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.788520 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.788308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-oauth-config\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.788520 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.788438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-config\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.788520 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.788480 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-oauth-serving-cert\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.788682 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.788544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-service-ca\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.889483 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.889449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhlg\" (UniqueName: \"kubernetes.io/projected/3de746a5-fc32-4c55-97d4-cb55fe39360b-kube-api-access-5hhlg\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.889483 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.889487 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-serving-cert\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.889707 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.889508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-oauth-config\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.889707 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.889554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-config\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.889707 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.889574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-oauth-serving-cert\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.889707 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.889604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-service-ca\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.890366 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.890338 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-oauth-serving-cert\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.890487 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.890421 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-config\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.890553 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.890521 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-service-ca\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.892576 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.892551 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-oauth-config\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.892708 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.892577 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-serving-cert\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.901546 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.901521 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhlg\" (UniqueName: \"kubernetes.io/projected/3de746a5-fc32-4c55-97d4-cb55fe39360b-kube-api-access-5hhlg\") pod \"console-68cc7f86db-2vvsf\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:12.920140 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.920106 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-2f4j7"] Sep 05 19:46:12.923298 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.923275 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:12.925967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.925942 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 05 19:46:12.926204 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.926146 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 05 19:46:12.926330 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.926233 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 05 19:46:12.926330 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.926292 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-krlzz\"" Sep 05 19:46:12.926330 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.926240 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 05 19:46:12.926549 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.926533 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 05 19:46:12.934056 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.934029 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-2f4j7"] Sep 05 19:46:12.957353 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:12.957307 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:13.090765 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.090723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/392a4d0a-31cb-43f7-9a22-695c9a54ad02-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.090945 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.090778 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/392a4d0a-31cb-43f7-9a22-695c9a54ad02-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.091008 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.090966 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/392a4d0a-31cb-43f7-9a22-695c9a54ad02-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.091075 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.091012 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dbs7\" (UniqueName: \"kubernetes.io/projected/392a4d0a-31cb-43f7-9a22-695c9a54ad02-kube-api-access-4dbs7\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.192556 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.192462 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/392a4d0a-31cb-43f7-9a22-695c9a54ad02-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.192717 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.192571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dbs7\" (UniqueName: \"kubernetes.io/projected/392a4d0a-31cb-43f7-9a22-695c9a54ad02-kube-api-access-4dbs7\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.192717 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:46:13.192614 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 05 19:46:13.192717 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.192631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/392a4d0a-31cb-43f7-9a22-695c9a54ad02-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.192717 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.192667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/392a4d0a-31cb-43f7-9a22-695c9a54ad02-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.192717 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:46:13.192685 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/392a4d0a-31cb-43f7-9a22-695c9a54ad02-prometheus-operator-tls podName:392a4d0a-31cb-43f7-9a22-695c9a54ad02 nodeName:}" failed. No retries permitted until 2026-09-05 19:46:13.692661466 +0000 UTC m=+194.177962322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/392a4d0a-31cb-43f7-9a22-695c9a54ad02-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-2f4j7" (UID: "392a4d0a-31cb-43f7-9a22-695c9a54ad02") : secret "prometheus-operator-tls" not found Sep 05 19:46:13.193376 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.193352 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/392a4d0a-31cb-43f7-9a22-695c9a54ad02-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.195384 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.195359 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/392a4d0a-31cb-43f7-9a22-695c9a54ad02-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.201340 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.201318 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dbs7\" (UniqueName: \"kubernetes.io/projected/392a4d0a-31cb-43f7-9a22-695c9a54ad02-kube-api-access-4dbs7\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.240237 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.240196 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68cc7f86db-2vvsf"] Sep 05 19:46:13.245477 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:46:13.245450 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de746a5_fc32_4c55_97d4_cb55fe39360b.slice/crio-e2e2fba2f8c114ebfb4976e28f7de0af91d3c06bc90e95d8ba22423d44396d6f WatchSource:0}: Error finding container e2e2fba2f8c114ebfb4976e28f7de0af91d3c06bc90e95d8ba22423d44396d6f: Status 404 returned error can't find the container with id e2e2fba2f8c114ebfb4976e28f7de0af91d3c06bc90e95d8ba22423d44396d6f Sep 05 19:46:13.698216 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.698052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/392a4d0a-31cb-43f7-9a22-695c9a54ad02-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.701281 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.701252 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/392a4d0a-31cb-43f7-9a22-695c9a54ad02-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2f4j7\" (UID: \"392a4d0a-31cb-43f7-9a22-695c9a54ad02\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:13.704221 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.704188 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-85hn8" event={"ID":"a9891d97-10d1-45d7-92c2-b2e28465c386","Type":"ContainerStarted","Data":"a0689e9a6d700f7df149b23afee9ad11cdfedbe1187d1ce5ae87286df3c46321"} Sep 05 19:46:13.705945 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.705829 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68cc7f86db-2vvsf" event={"ID":"3de746a5-fc32-4c55-97d4-cb55fe39360b","Type":"ContainerStarted","Data":"e2e2fba2f8c114ebfb4976e28f7de0af91d3c06bc90e95d8ba22423d44396d6f"} Sep 05 19:46:13.835149 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:13.835014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" Sep 05 19:46:14.133520 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:14.133425 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-2f4j7"] Sep 05 19:46:14.448733 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:46:14.448637 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392a4d0a_31cb_43f7_9a22_695c9a54ad02.slice/crio-3212ba824c742e03c2d3fbbee9b56ab62f126a87bab335277793e62c1b536a35 WatchSource:0}: Error finding container 3212ba824c742e03c2d3fbbee9b56ab62f126a87bab335277793e62c1b536a35: Status 404 returned error can't find the container with id 3212ba824c742e03c2d3fbbee9b56ab62f126a87bab335277793e62c1b536a35 Sep 05 19:46:14.710712 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:14.710629 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" event={"ID":"392a4d0a-31cb-43f7-9a22-695c9a54ad02","Type":"ContainerStarted","Data":"3212ba824c742e03c2d3fbbee9b56ab62f126a87bab335277793e62c1b536a35"} Sep 05 19:46:16.475249 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.475211 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-556645967-fx9jh"] Sep 05 19:46:16.478426 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.478405 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.487749 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.487707 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 05 19:46:16.489969 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.489946 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556645967-fx9jh"] Sep 05 19:46:16.623818 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.623770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-service-ca\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.623998 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.623831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-oauth-serving-cert\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.623998 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.623928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7rp\" (UniqueName: \"kubernetes.io/projected/c5aca7fb-4297-49f5-995a-1ca5ba800724-kube-api-access-cf7rp\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.623998 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.623969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-serving-cert\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.624165 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.623999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-oauth-config\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.624165 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.624089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-trusted-ca-bundle\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.624165 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.624123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-config\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.725312 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.725223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-trusted-ca-bundle\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.725312 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.725262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-config\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.725561 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.725339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-service-ca\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.725561 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.725368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-oauth-serving-cert\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.725561 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.725427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7rp\" (UniqueName: \"kubernetes.io/projected/c5aca7fb-4297-49f5-995a-1ca5ba800724-kube-api-access-cf7rp\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.725561 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.725471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-serving-cert\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.725561 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.725544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-oauth-config\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.726317 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.726289 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-trusted-ca-bundle\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.726406 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.726289 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-oauth-serving-cert\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.726406 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.726373 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-service-ca\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.726494 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.726452 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-config\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.728137 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.728114 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-serving-cert\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.728259 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.728179 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-oauth-config\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.734617 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.734595 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7rp\" (UniqueName: \"kubernetes.io/projected/c5aca7fb-4297-49f5-995a-1ca5ba800724-kube-api-access-cf7rp\") pod \"console-556645967-fx9jh\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:16.789777 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:16.789731 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:22.902209 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:22.902123 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556645967-fx9jh"] Sep 05 19:46:22.969187 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:46:22.969133 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5aca7fb_4297_49f5_995a_1ca5ba800724.slice/crio-b9d90257e097bf205ff20965ee44f0fd0c879cf2fe385cdb6ed27a6e224a4e7d WatchSource:0}: Error finding container b9d90257e097bf205ff20965ee44f0fd0c879cf2fe385cdb6ed27a6e224a4e7d: Status 404 returned error can't find the container with id b9d90257e097bf205ff20965ee44f0fd0c879cf2fe385cdb6ed27a6e224a4e7d Sep 05 19:46:23.746541 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.746498 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-n5qrf" event={"ID":"7ae64473-c8b8-42cf-bc67-c7d67f744d2c","Type":"ContainerStarted","Data":"5f4e02b64dad2d3ad50a8b642713f41a289a6ae34ecc94f3634a3772f8648f5b"} Sep 05 19:46:23.746756 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.746602 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-n5qrf" Sep 05 19:46:23.748765 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.748681 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556645967-fx9jh" event={"ID":"c5aca7fb-4297-49f5-995a-1ca5ba800724","Type":"ContainerStarted","Data":"1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3"} Sep 05 19:46:23.748765 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.748736 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556645967-fx9jh" event={"ID":"c5aca7fb-4297-49f5-995a-1ca5ba800724","Type":"ContainerStarted","Data":"b9d90257e097bf205ff20965ee44f0fd0c879cf2fe385cdb6ed27a6e224a4e7d"} Sep 05 19:46:23.750693 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.750668 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" event={"ID":"392a4d0a-31cb-43f7-9a22-695c9a54ad02","Type":"ContainerStarted","Data":"d9bada8377ca92e002c5eadc8a1115d45b9824c5ef11f01319412b8c4dcfcf05"} Sep 05 19:46:23.750816 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.750718 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" event={"ID":"392a4d0a-31cb-43f7-9a22-695c9a54ad02","Type":"ContainerStarted","Data":"5a0cc9f4d0c86d94b4e7b6f0b6b6d0107adc3425a95df501498f506d4796e924"} Sep 05 19:46:23.752996 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.752972 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-85hn8" event={"ID":"a9891d97-10d1-45d7-92c2-b2e28465c386","Type":"ContainerStarted","Data":"c847ada96af8b011ffcf8fa8070de7ea609fc325b947690c276becfa9be91fa4"} Sep 05 19:46:23.754837 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.754815 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68cc7f86db-2vvsf" event={"ID":"3de746a5-fc32-4c55-97d4-cb55fe39360b","Type":"ContainerStarted","Data":"5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6"} Sep 05 19:46:23.760036 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.760016 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-n5qrf" Sep 05 19:46:23.768195 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.768140 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-n5qrf" podStartSLOduration=2.054902764 podStartE2EDuration="19.768124208s" totalImagesPullingTime="17.713221444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:23.767719003 +0000 UTC m=+204.253019877" watchObservedRunningTime="2026-09-05 19:46:23.768124208 +0000 UTC m=+204.253425082" Sep 05 19:46:23.788175 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.788129 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-85hn8" podStartSLOduration=34.502873327 podStartE2EDuration="44.788110411s" totalImagesPullingTime="10.285237084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:45:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:23.787701895 +0000 UTC m=+204.273002770" watchObservedRunningTime="2026-09-05 19:46:23.788110411 +0000 UTC m=+204.273411286" Sep 05 19:46:23.831350 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.831292 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68cc7f86db-2vvsf" podStartSLOduration=2.110469037 podStartE2EDuration="11.831268973s" totalImagesPullingTime="9.720799936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:23.830080777 +0000 UTC m=+204.315381651" watchObservedRunningTime="2026-09-05 19:46:23.831268973 +0000 UTC m=+204.316569847" Sep 05 19:46:23.850404 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.850344 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-2f4j7" podStartSLOduration=3.279817231 podStartE2EDuration="11.850321903s" totalImagesPullingTime="8.570504672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:23.849515555 +0000 UTC m=+204.334816681" watchObservedRunningTime="2026-09-05 19:46:23.850321903 +0000 UTC m=+204.335622777" Sep 05 19:46:23.870286 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:23.870231 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-556645967-fx9jh" podStartSLOduration=7.870209182 podStartE2EDuration="7.870209182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:23.869116775 +0000 UTC m=+204.354417683" watchObservedRunningTime="2026-09-05 19:46:23.870209182 +0000 UTC m=+204.355510120" Sep 05 19:46:25.308022 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.307984 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-298md"] Sep 05 19:46:25.332102 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.332063 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.335686 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.335652 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 05 19:46:25.335875 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.335685 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4sdpt\"" Sep 05 19:46:25.335875 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.335653 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 05 19:46:25.335998 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.335933 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 05 19:46:25.400422 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.400380 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.400600 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.400451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-textfile\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.400600 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.400497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.400600 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.400544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-wtmp\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.400600 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.400576 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4f4t\" (UniqueName: \"kubernetes.io/projected/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-kube-api-access-d4f4t\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.400778 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.400614 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-tls\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.400778 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.400698 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-root\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.400778 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.400748 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-sys\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.400910 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.400787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-metrics-client-ca\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.501327 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-root\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.501497 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-sys\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.501497 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-metrics-client-ca\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.501497 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.501497 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501426 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-root\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.501497 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-textfile\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.501497 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501486 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-sys\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.501849 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.501849 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-wtmp\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.501849 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4f4t\" (UniqueName: \"kubernetes.io/projected/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-kube-api-access-d4f4t\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.501849 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-tls\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.501849 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501759 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-wtmp\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.502105 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.501955 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-textfile\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.502219 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.502198 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.502285 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.502237 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-metrics-client-ca\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.504477 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.504450 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.504595 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.504540 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-node-exporter-tls\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.511969 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.511947 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4f4t\" (UniqueName: \"kubernetes.io/projected/a1254285-9e7d-4ad2-b3c7-5d446e29f0fe-kube-api-access-d4f4t\") pod \"node-exporter-298md\" (UID: \"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe\") " pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.644179 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.644096 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-298md" Sep 05 19:46:25.679507 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:46:25.679465 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1254285_9e7d_4ad2_b3c7_5d446e29f0fe.slice/crio-ed9af308951f042c96a788a138e2e92415cb805ce9c65652746a9c9c94c0e5e4 WatchSource:0}: Error finding container ed9af308951f042c96a788a138e2e92415cb805ce9c65652746a9c9c94c0e5e4: Status 404 returned error can't find the container with id ed9af308951f042c96a788a138e2e92415cb805ce9c65652746a9c9c94c0e5e4 Sep 05 19:46:25.764090 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:25.764046 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-298md" event={"ID":"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe","Type":"ContainerStarted","Data":"ed9af308951f042c96a788a138e2e92415cb805ce9c65652746a9c9c94c0e5e4"} Sep 05 19:46:26.792182 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:26.792097 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:26.793342 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:26.793234 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:26.798444 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:26.798419 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:27.166197 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.166120 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b7464dff9-hhskf"] Sep 05 19:46:27.166447 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:46:27.166411 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" podUID="ddc842b4-0f78-458c-ab9a-0f101d479c0b" Sep 05 19:46:27.774889 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.774846 2578 generic.go:356] "Generic (PLEG): container finished" podID="a1254285-9e7d-4ad2-b3c7-5d446e29f0fe" containerID="7116d291078f8e9467f813fe871c03f6b4e026f3a52d6fcf8bb6577955fd3182" exitCode=0 Sep 05 19:46:27.775089 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.774942 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-298md" event={"ID":"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe","Type":"ContainerDied","Data":"7116d291078f8e9467f813fe871c03f6b4e026f3a52d6fcf8bb6577955fd3182"} Sep 05 19:46:27.775089 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.774994 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:46:27.779988 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.779917 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-556645967-fx9jh" Sep 05 19:46:27.798145 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.798117 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:46:27.824342 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.824314 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ddc842b4-0f78-458c-ab9a-0f101d479c0b-image-registry-private-configuration\") pod \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " Sep 05 19:46:27.824475 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.824369 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ddc842b4-0f78-458c-ab9a-0f101d479c0b-installation-pull-secrets\") pod \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " Sep 05 19:46:27.824475 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.824428 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-bound-sa-token\") pod \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " Sep 05 19:46:27.824475 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.824455 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gds8j\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-kube-api-access-gds8j\") pod \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " Sep 05 19:46:27.824621 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.824493 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-certificates\") pod \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " Sep 05 19:46:27.824621 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.824555 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ddc842b4-0f78-458c-ab9a-0f101d479c0b-ca-trust-extracted\") pod \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " Sep 05 19:46:27.824621 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.824602 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ddc842b4-0f78-458c-ab9a-0f101d479c0b-trusted-ca\") pod \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\" (UID: \"ddc842b4-0f78-458c-ab9a-0f101d479c0b\") " Sep 05 19:46:27.826977 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.825076 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ddc842b4-0f78-458c-ab9a-0f101d479c0b" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:46:27.826977 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.825488 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc842b4-0f78-458c-ab9a-0f101d479c0b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ddc842b4-0f78-458c-ab9a-0f101d479c0b" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 19:46:27.826977 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.825885 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc842b4-0f78-458c-ab9a-0f101d479c0b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ddc842b4-0f78-458c-ab9a-0f101d479c0b" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:46:27.829267 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.829219 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc842b4-0f78-458c-ab9a-0f101d479c0b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ddc842b4-0f78-458c-ab9a-0f101d479c0b" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:46:27.829372 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.829343 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ddc842b4-0f78-458c-ab9a-0f101d479c0b" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:46:27.831606 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.831578 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc842b4-0f78-458c-ab9a-0f101d479c0b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ddc842b4-0f78-458c-ab9a-0f101d479c0b" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:46:27.831606 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.831592 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-kube-api-access-gds8j" (OuterVolumeSpecName: "kube-api-access-gds8j") pod "ddc842b4-0f78-458c-ab9a-0f101d479c0b" (UID: "ddc842b4-0f78-458c-ab9a-0f101d479c0b"). InnerVolumeSpecName "kube-api-access-gds8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:46:27.849571 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.849539 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68cc7f86db-2vvsf"] Sep 05 19:46:27.925617 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.925592 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ddc842b4-0f78-458c-ab9a-0f101d479c0b-trusted-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:27.925738 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.925620 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ddc842b4-0f78-458c-ab9a-0f101d479c0b-image-registry-private-configuration\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:27.925738 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.925636 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ddc842b4-0f78-458c-ab9a-0f101d479c0b-installation-pull-secrets\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:27.925738 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.925649 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-bound-sa-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:27.925738 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.925661 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gds8j\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-kube-api-access-gds8j\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:27.925738 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.925673 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-certificates\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:27.925738 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:27.925685 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ddc842b4-0f78-458c-ab9a-0f101d479c0b-ca-trust-extracted\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:28.781885 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:28.781828 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-298md" event={"ID":"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe","Type":"ContainerStarted","Data":"01df93721aa732bd843483dba596d4a2a23003eb2706820aa0dd9157b7fa3899"} Sep 05 19:46:28.782135 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:28.781908 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-298md" event={"ID":"a1254285-9e7d-4ad2-b3c7-5d446e29f0fe","Type":"ContainerStarted","Data":"3b6adefbc0fdf119f7ff298d0e25aee4cf004ba73d43bda5fbb38be0f835c40d"} Sep 05 19:46:28.782245 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:28.782225 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7464dff9-hhskf" Sep 05 19:46:28.807308 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:28.807253 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-298md" podStartSLOduration=2.10494464 podStartE2EDuration="3.807236122s" totalImagesPullingTime="1.611347294s" totalInitContainerRuntime="90.944188ms" isStatefulPod=true podCreationTimestamp="2026-09-05 19:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:28.805596831 +0000 UTC m=+209.290897717" watchObservedRunningTime="2026-09-05 19:46:28.807236122 +0000 UTC m=+209.292536995" Sep 05 19:46:28.847647 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:28.847615 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b7464dff9-hhskf"] Sep 05 19:46:28.851779 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:28.851753 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b7464dff9-hhskf"] Sep 05 19:46:28.934000 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:28.933966 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddc842b4-0f78-458c-ab9a-0f101d479c0b-registry-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:29.605297 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.605261 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr"] Sep 05 19:46:29.639182 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.639144 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr"] Sep 05 19:46:29.639362 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.639280 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.644259 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.644234 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mfrkt\"" Sep 05 19:46:29.644438 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.644417 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8966s1t6c3sra\"" Sep 05 19:46:29.644550 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.644464 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 05 19:46:29.644550 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.644333 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 05 19:46:29.644668 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.644372 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 05 19:46:29.644668 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.644273 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 05 19:46:29.742240 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.742204 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42dd5b05-f773-4a2f-96ba-cf5217e3841d-secret-metrics-server-tls\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.742421 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.742254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42dd5b05-f773-4a2f-96ba-cf5217e3841d-audit-log\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.742421 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.742271 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42dd5b05-f773-4a2f-96ba-cf5217e3841d-metrics-server-audit-profiles\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.742421 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.742298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42dd5b05-f773-4a2f-96ba-cf5217e3841d-client-ca-bundle\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.742421 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.742359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42dd5b05-f773-4a2f-96ba-cf5217e3841d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.742421 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.742399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlt5k\" (UniqueName: \"kubernetes.io/projected/42dd5b05-f773-4a2f-96ba-cf5217e3841d-kube-api-access-tlt5k\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.742670 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.742458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42dd5b05-f773-4a2f-96ba-cf5217e3841d-secret-metrics-server-client-certs\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.844061 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.844024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42dd5b05-f773-4a2f-96ba-cf5217e3841d-secret-metrics-server-client-certs\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.844540 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.844511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42dd5b05-f773-4a2f-96ba-cf5217e3841d-secret-metrics-server-tls\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.844612 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.844561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42dd5b05-f773-4a2f-96ba-cf5217e3841d-audit-log\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.844612 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.844590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42dd5b05-f773-4a2f-96ba-cf5217e3841d-metrics-server-audit-profiles\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.844714 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.844657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42dd5b05-f773-4a2f-96ba-cf5217e3841d-client-ca-bundle\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.844767 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.844710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42dd5b05-f773-4a2f-96ba-cf5217e3841d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.844767 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.844739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlt5k\" (UniqueName: \"kubernetes.io/projected/42dd5b05-f773-4a2f-96ba-cf5217e3841d-kube-api-access-tlt5k\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.845040 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.845013 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42dd5b05-f773-4a2f-96ba-cf5217e3841d-audit-log\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.845506 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.845480 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42dd5b05-f773-4a2f-96ba-cf5217e3841d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.845722 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.845695 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42dd5b05-f773-4a2f-96ba-cf5217e3841d-metrics-server-audit-profiles\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.847236 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.847203 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42dd5b05-f773-4a2f-96ba-cf5217e3841d-secret-metrics-server-client-certs\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.847451 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.847428 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42dd5b05-f773-4a2f-96ba-cf5217e3841d-secret-metrics-server-tls\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.847579 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.847560 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42dd5b05-f773-4a2f-96ba-cf5217e3841d-client-ca-bundle\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.856264 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.856210 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlt5k\" (UniqueName: \"kubernetes.io/projected/42dd5b05-f773-4a2f-96ba-cf5217e3841d-kube-api-access-tlt5k\") pod \"metrics-server-7cbf6b79f5-cqqhr\" (UID: \"42dd5b05-f773-4a2f-96ba-cf5217e3841d\") " pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:29.951737 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:29.951696 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:30.012161 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.012122 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddc842b4-0f78-458c-ab9a-0f101d479c0b" path="/var/lib/kubelet/pods/ddc842b4-0f78-458c-ab9a-0f101d479c0b/volumes" Sep 05 19:46:30.068389 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.067911 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk"] Sep 05 19:46:30.103218 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.103174 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk"] Sep 05 19:46:30.103386 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.103298 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" Sep 05 19:46:30.106197 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.106168 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 05 19:46:30.106386 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.106338 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-d5qvl\"" Sep 05 19:46:30.147380 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.147340 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df4f33b8-cae3-4624-b9ef-0c4cd2ba4951-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fxzkk\" (UID: \"df4f33b8-cae3-4624-b9ef-0c4cd2ba4951\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" Sep 05 19:46:30.147556 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.147474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxvq\" (UniqueName: \"kubernetes.io/projected/df4f33b8-cae3-4624-b9ef-0c4cd2ba4951-kube-api-access-tlxvq\") pod \"monitoring-plugin-85bbbbbf4-fxzkk\" (UID: \"df4f33b8-cae3-4624-b9ef-0c4cd2ba4951\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" Sep 05 19:46:30.248088 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.248048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df4f33b8-cae3-4624-b9ef-0c4cd2ba4951-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fxzkk\" (UID: \"df4f33b8-cae3-4624-b9ef-0c4cd2ba4951\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" Sep 05 19:46:30.248270 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.248148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxvq\" (UniqueName: \"kubernetes.io/projected/df4f33b8-cae3-4624-b9ef-0c4cd2ba4951-kube-api-access-tlxvq\") pod \"monitoring-plugin-85bbbbbf4-fxzkk\" (UID: \"df4f33b8-cae3-4624-b9ef-0c4cd2ba4951\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" Sep 05 19:46:30.248270 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:46:30.248213 2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 05 19:46:30.248381 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:46:30.248293 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4f33b8-cae3-4624-b9ef-0c4cd2ba4951-monitoring-plugin-cert podName:df4f33b8-cae3-4624-b9ef-0c4cd2ba4951 nodeName:}" failed. No retries permitted until 2026-09-05 19:46:30.748274098 +0000 UTC m=+211.233574949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/df4f33b8-cae3-4624-b9ef-0c4cd2ba4951-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-fxzkk" (UID: "df4f33b8-cae3-4624-b9ef-0c4cd2ba4951") : secret "monitoring-plugin-cert" not found Sep 05 19:46:30.258296 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.258266 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxvq\" (UniqueName: \"kubernetes.io/projected/df4f33b8-cae3-4624-b9ef-0c4cd2ba4951-kube-api-access-tlxvq\") pod \"monitoring-plugin-85bbbbbf4-fxzkk\" (UID: \"df4f33b8-cae3-4624-b9ef-0c4cd2ba4951\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" Sep 05 19:46:30.267194 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.267160 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr"] Sep 05 19:46:30.271484 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:46:30.271454 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dd5b05_f773_4a2f_96ba_cf5217e3841d.slice/crio-f04fc5ab49c24c1de1d0ec3a93d85602aa79603d441953899c7666f76bd8f856 WatchSource:0}: Error finding container f04fc5ab49c24c1de1d0ec3a93d85602aa79603d441953899c7666f76bd8f856: Status 404 returned error can't find the container with id f04fc5ab49c24c1de1d0ec3a93d85602aa79603d441953899c7666f76bd8f856 Sep 05 19:46:30.753520 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.753476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df4f33b8-cae3-4624-b9ef-0c4cd2ba4951-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fxzkk\" (UID: \"df4f33b8-cae3-4624-b9ef-0c4cd2ba4951\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" Sep 05 19:46:30.756323 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.756293 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df4f33b8-cae3-4624-b9ef-0c4cd2ba4951-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fxzkk\" (UID: \"df4f33b8-cae3-4624-b9ef-0c4cd2ba4951\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" Sep 05 19:46:30.792811 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:30.792758 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" event={"ID":"42dd5b05-f773-4a2f-96ba-cf5217e3841d","Type":"ContainerStarted","Data":"f04fc5ab49c24c1de1d0ec3a93d85602aa79603d441953899c7666f76bd8f856"} Sep 05 19:46:31.014615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.014523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" Sep 05 19:46:31.322731 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.322567 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk"] Sep 05 19:46:31.608717 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.608600 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 05 19:46:31.642573 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.642521 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 05 19:46:31.642746 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.642704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.646651 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.646614 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 05 19:46:31.647589 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.646836 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 05 19:46:31.647589 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.646772 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 05 19:46:31.647589 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.647051 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 05 19:46:31.647589 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.647242 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 05 19:46:31.647589 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.647283 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9pwdw\"" Sep 05 19:46:31.647981 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.647768 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 05 19:46:31.648093 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.648067 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 05 19:46:31.648232 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.648148 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 05 19:46:31.648307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.648285 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 05 19:46:31.648455 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.648407 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3hafphn87bks3\"" Sep 05 19:46:31.648757 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.648668 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 05 19:46:31.649405 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.649386 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 05 19:46:31.651392 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.649562 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 05 19:46:31.653351 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.653270 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 05 19:46:31.655967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.655878 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 05 19:46:31.763161 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763121 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef17468b-7f9b-449c-a708-bed11f271a8e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef17468b-7f9b-449c-a708-bed11f271a8e-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763317 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763349 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763681 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763681 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763681 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763681 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763681 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763568 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763681 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763681 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.763681 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763681 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.764107 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.764107 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763730 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z45r\" (UniqueName: \"kubernetes.io/projected/ef17468b-7f9b-449c-a708-bed11f271a8e-kube-api-access-6z45r\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.764107 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763759 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-config\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.764107 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.763783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.799689 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.799636 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" event={"ID":"df4f33b8-cae3-4624-b9ef-0c4cd2ba4951","Type":"ContainerStarted","Data":"ae7e6e3f859f31d2b21536bc625918b9e9f8374901c09bc565aed2b2aaa226c5"} Sep 05 19:46:31.865304 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865304 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865521 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865521 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865521 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865521 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865521 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865480 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865521 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865840 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865539 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z45r\" (UniqueName: \"kubernetes.io/projected/ef17468b-7f9b-449c-a708-bed11f271a8e-kube-api-access-6z45r\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865840 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-config\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865840 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865840 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865840 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865840 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865840 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865840 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef17468b-7f9b-449c-a708-bed11f271a8e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.865840 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef17468b-7f9b-449c-a708-bed11f271a8e-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.866445 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.866445 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.866445 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.866445 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.866302 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.866642 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.866555 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.868035 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.867323 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.868035 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.865677 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.869612 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.869592 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.872226 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.872207 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.874329 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.873861 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.874329 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.874092 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.874329 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.874289 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.875957 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.875938 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-config\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.878505 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.878462 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.878877 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.878768 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.879145 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.879110 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.880159 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.879899 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.884912 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.882753 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.885865 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.885817 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef17468b-7f9b-449c-a708-bed11f271a8e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.886407 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.886065 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef17468b-7f9b-449c-a708-bed11f271a8e-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.886407 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.886180 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.886407 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.886373 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.886851 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.886830 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z45r\" (UniqueName: \"kubernetes.io/projected/ef17468b-7f9b-449c-a708-bed11f271a8e-kube-api-access-6z45r\") pod \"prometheus-k8s-0\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:31.960206 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:31.960158 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:32.448931 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:32.448860 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 05 19:46:32.463012 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:46:32.462980 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef17468b_7f9b_449c_a708_bed11f271a8e.slice/crio-006eb5f822d0db1499639759a18f8baa41af2213eb54e609c9bd469b8ed8e24b WatchSource:0}: Error finding container 006eb5f822d0db1499639759a18f8baa41af2213eb54e609c9bd469b8ed8e24b: Status 404 returned error can't find the container with id 006eb5f822d0db1499639759a18f8baa41af2213eb54e609c9bd469b8ed8e24b Sep 05 19:46:32.807569 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:32.807497 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" event={"ID":"42dd5b05-f773-4a2f-96ba-cf5217e3841d","Type":"ContainerStarted","Data":"85a37e18e1ebff382ca953778b243a6e1d4f0417197eb7b589aabd1a6e3dd80a"} Sep 05 19:46:32.809111 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:32.809069 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerStarted","Data":"006eb5f822d0db1499639759a18f8baa41af2213eb54e609c9bd469b8ed8e24b"} Sep 05 19:46:32.832473 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:32.832407 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" podStartSLOduration=1.614174647 podStartE2EDuration="3.832385546s" totalImagesPullingTime="2.218210899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:32.832212471 +0000 UTC m=+213.317513346" watchObservedRunningTime="2026-09-05 19:46:32.832385546 +0000 UTC m=+213.317686420" Sep 05 19:46:32.958299 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:32.958264 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:34.817444 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:34.817401 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" event={"ID":"df4f33b8-cae3-4624-b9ef-0c4cd2ba4951","Type":"ContainerStarted","Data":"f01c520626c2fb8ae0734ee51edd2f8ee6f33c54dfdf22daa5babbe2ee0ec9d8"} Sep 05 19:46:34.817953 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:34.817570 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" Sep 05 19:46:34.819154 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:34.819131 2578 generic.go:356] "Generic (PLEG): container finished" podID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerID="1782e9465eec1edeb32bfc41ce3d06849964cbe9a8dd61d4294c2928185a4709" exitCode=0 Sep 05 19:46:34.819274 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:34.819192 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerDied","Data":"1782e9465eec1edeb32bfc41ce3d06849964cbe9a8dd61d4294c2928185a4709"} Sep 05 19:46:34.823781 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:34.823758 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" Sep 05 19:46:34.836237 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:34.836187 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fxzkk" podStartSLOduration=1.996623753 podStartE2EDuration="4.836169973s" totalImagesPullingTime="2.83954622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:34.834918783 +0000 UTC m=+215.320219656" watchObservedRunningTime="2026-09-05 19:46:34.836169973 +0000 UTC m=+215.321470845" Sep 05 19:46:38.837995 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:38.837959 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerStarted","Data":"1a082482174d56b19f7b31f26d69a762c084458448695d9735b2e4b35509c14c"} Sep 05 19:46:38.838353 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:38.838058 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerStarted","Data":"357081889e3e02d9977fbc1a3b02e53d51fcbda463e71577d5849f98860d5fae"} Sep 05 19:46:41.852403 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:41.852365 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerStarted","Data":"08eccb3c53e839c0bafefefd74f51f9811951221aa9f13cf57df0ca238047075"} Sep 05 19:46:41.852964 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:41.852433 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerStarted","Data":"b37fe8bcef27d7b67c9cef84e4d71fad3d9cdf4f3a4ae20b86c9c84cd7fac7b2"} Sep 05 19:46:41.852964 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:41.852459 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerStarted","Data":"54adaefad14811f007a1528c7c55589e97aa9f2728d18224450faa467ad58410"} Sep 05 19:46:41.852964 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:41.852483 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerStarted","Data":"d0999d74a64d79a88c602eb47bfe48981d55d6365ad61df97cca5ef9e10401a4"} Sep 05 19:46:41.913756 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:41.913696 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.126235568 podStartE2EDuration="10.913680473s" totalImagesPullingTime="7.762001539s" totalInitContainerRuntime="25.443366ms" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:41.911614195 +0000 UTC m=+222.396915116" watchObservedRunningTime="2026-09-05 19:46:41.913680473 +0000 UTC m=+222.398981345" Sep 05 19:46:41.961233 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:41.961198 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:46:41.967995 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:41.967504 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556645967-fx9jh"] Sep 05 19:46:49.952430 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:49.952386 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:49.952926 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:49.952465 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:46:52.875282 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:52.875216 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68cc7f86db-2vvsf" podUID="3de746a5-fc32-4c55-97d4-cb55fe39360b" containerName="console" containerID="cri-o://5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6" gracePeriod=15 Sep 05 19:46:53.122642 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.122611 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:53.264680 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.264650 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hhlg\" (UniqueName: \"kubernetes.io/projected/3de746a5-fc32-4c55-97d4-cb55fe39360b-kube-api-access-5hhlg\") pod \"3de746a5-fc32-4c55-97d4-cb55fe39360b\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " Sep 05 19:46:53.264872 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.264711 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-service-ca\") pod \"3de746a5-fc32-4c55-97d4-cb55fe39360b\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " Sep 05 19:46:53.264872 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.264756 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-serving-cert\") pod \"3de746a5-fc32-4c55-97d4-cb55fe39360b\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " Sep 05 19:46:53.264872 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.264786 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-oauth-config\") pod \"3de746a5-fc32-4c55-97d4-cb55fe39360b\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " Sep 05 19:46:53.264872 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.264836 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-config\") pod \"3de746a5-fc32-4c55-97d4-cb55fe39360b\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " Sep 05 19:46:53.265058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.264878 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-oauth-serving-cert\") pod \"3de746a5-fc32-4c55-97d4-cb55fe39360b\" (UID: \"3de746a5-fc32-4c55-97d4-cb55fe39360b\") " Sep 05 19:46:53.265216 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.265184 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-service-ca" (OuterVolumeSpecName: "service-ca") pod "3de746a5-fc32-4c55-97d4-cb55fe39360b" (UID: "3de746a5-fc32-4c55-97d4-cb55fe39360b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:46:53.265310 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.265237 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-config" (OuterVolumeSpecName: "console-config") pod "3de746a5-fc32-4c55-97d4-cb55fe39360b" (UID: "3de746a5-fc32-4c55-97d4-cb55fe39360b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:46:53.265357 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.265297 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3de746a5-fc32-4c55-97d4-cb55fe39360b" (UID: "3de746a5-fc32-4c55-97d4-cb55fe39360b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:46:53.267097 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.267065 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3de746a5-fc32-4c55-97d4-cb55fe39360b" (UID: "3de746a5-fc32-4c55-97d4-cb55fe39360b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:46:53.267158 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.267131 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3de746a5-fc32-4c55-97d4-cb55fe39360b" (UID: "3de746a5-fc32-4c55-97d4-cb55fe39360b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:46:53.267234 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.267215 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de746a5-fc32-4c55-97d4-cb55fe39360b-kube-api-access-5hhlg" (OuterVolumeSpecName: "kube-api-access-5hhlg") pod "3de746a5-fc32-4c55-97d4-cb55fe39360b" (UID: "3de746a5-fc32-4c55-97d4-cb55fe39360b"). InnerVolumeSpecName "kube-api-access-5hhlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:46:53.366189 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.366148 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:53.366189 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.366180 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-serving-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:53.366189 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.366190 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-oauth-config\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:53.366189 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.366198 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-console-config\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:53.366435 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.366207 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3de746a5-fc32-4c55-97d4-cb55fe39360b-oauth-serving-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:53.366435 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.366215 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hhlg\" (UniqueName: \"kubernetes.io/projected/3de746a5-fc32-4c55-97d4-cb55fe39360b-kube-api-access-5hhlg\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:46:53.900054 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.900018 2578 generic.go:356] "Generic (PLEG): container finished" podID="3de746a5-fc32-4c55-97d4-cb55fe39360b" containerID="5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6" exitCode=0 Sep 05 19:46:53.900451 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.900054 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68cc7f86db-2vvsf" event={"ID":"3de746a5-fc32-4c55-97d4-cb55fe39360b","Type":"ContainerDied","Data":"5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6"} Sep 05 19:46:53.900451 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.900084 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68cc7f86db-2vvsf" Sep 05 19:46:53.900451 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.900110 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68cc7f86db-2vvsf" event={"ID":"3de746a5-fc32-4c55-97d4-cb55fe39360b","Type":"ContainerDied","Data":"e2e2fba2f8c114ebfb4976e28f7de0af91d3c06bc90e95d8ba22423d44396d6f"} Sep 05 19:46:53.900451 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.900131 2578 scope.go:117] "RemoveContainer" containerID="5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6" Sep 05 19:46:53.909355 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.909335 2578 scope.go:117] "RemoveContainer" containerID="5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6" Sep 05 19:46:53.909703 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:46:53.909682 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6\": container with ID starting with 5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6 not found: ID does not exist" containerID="5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6" Sep 05 19:46:53.909768 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.909713 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6"} err="failed to get container status \"5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6\": rpc error: code = NotFound desc = could not find container \"5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6\": container with ID starting with 5bc131b1f0c83d8fbf3cc6578f71950371dafaf501039b76d3ac56f7947e7db6 not found: ID does not exist" Sep 05 19:46:53.921702 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.921668 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68cc7f86db-2vvsf"] Sep 05 19:46:53.927570 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:53.927540 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68cc7f86db-2vvsf"] Sep 05 19:46:54.013636 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:54.013601 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de746a5-fc32-4c55-97d4-cb55fe39360b" path="/var/lib/kubelet/pods/3de746a5-fc32-4c55-97d4-cb55fe39360b/volumes" Sep 05 19:46:58.924268 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:58.924164 2578 generic.go:356] "Generic (PLEG): container finished" podID="66922fa5-4d55-479f-b0b5-baef39b284f3" containerID="ff98802480a7b7f0ed2f769e8b450c0d3419c7962366e12c5886d1cd0e4e3f3d" exitCode=0 Sep 05 19:46:58.924268 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:58.924236 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" event={"ID":"66922fa5-4d55-479f-b0b5-baef39b284f3","Type":"ContainerDied","Data":"ff98802480a7b7f0ed2f769e8b450c0d3419c7962366e12c5886d1cd0e4e3f3d"} Sep 05 19:46:58.924788 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:58.924660 2578 scope.go:117] "RemoveContainer" containerID="ff98802480a7b7f0ed2f769e8b450c0d3419c7962366e12c5886d1cd0e4e3f3d" Sep 05 19:46:59.929851 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:46:59.929822 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-nptkb" event={"ID":"66922fa5-4d55-479f-b0b5-baef39b284f3","Type":"ContainerStarted","Data":"bb62273e52df52ba729d8284592e887d956fe722cac4abd088e9732cf3e5feec"} Sep 05 19:47:06.986958 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:06.986910 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-556645967-fx9jh" podUID="c5aca7fb-4297-49f5-995a-1ca5ba800724" containerName="console" containerID="cri-o://1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3" gracePeriod=15 Sep 05 19:47:07.251832 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.251741 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556645967-fx9jh" Sep 05 19:47:07.288705 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.288669 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-serving-cert\") pod \"c5aca7fb-4297-49f5-995a-1ca5ba800724\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " Sep 05 19:47:07.288705 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.288712 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-trusted-ca-bundle\") pod \"c5aca7fb-4297-49f5-995a-1ca5ba800724\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " Sep 05 19:47:07.288970 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.288752 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-oauth-config\") pod \"c5aca7fb-4297-49f5-995a-1ca5ba800724\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " Sep 05 19:47:07.288970 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.288787 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-service-ca\") pod \"c5aca7fb-4297-49f5-995a-1ca5ba800724\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " Sep 05 19:47:07.288970 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.288821 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-oauth-serving-cert\") pod \"c5aca7fb-4297-49f5-995a-1ca5ba800724\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " Sep 05 19:47:07.288970 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.288863 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-config\") pod \"c5aca7fb-4297-49f5-995a-1ca5ba800724\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " Sep 05 19:47:07.288970 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.288895 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf7rp\" (UniqueName: \"kubernetes.io/projected/c5aca7fb-4297-49f5-995a-1ca5ba800724-kube-api-access-cf7rp\") pod \"c5aca7fb-4297-49f5-995a-1ca5ba800724\" (UID: \"c5aca7fb-4297-49f5-995a-1ca5ba800724\") " Sep 05 19:47:07.289227 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.289151 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c5aca7fb-4297-49f5-995a-1ca5ba800724" (UID: "c5aca7fb-4297-49f5-995a-1ca5ba800724"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:47:07.289418 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.289388 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c5aca7fb-4297-49f5-995a-1ca5ba800724" (UID: "c5aca7fb-4297-49f5-995a-1ca5ba800724"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:47:07.289521 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.289460 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-config" (OuterVolumeSpecName: "console-config") pod "c5aca7fb-4297-49f5-995a-1ca5ba800724" (UID: "c5aca7fb-4297-49f5-995a-1ca5ba800724"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:47:07.289584 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.289562 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-service-ca" (OuterVolumeSpecName: "service-ca") pod "c5aca7fb-4297-49f5-995a-1ca5ba800724" (UID: "c5aca7fb-4297-49f5-995a-1ca5ba800724"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:47:07.290960 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.290928 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c5aca7fb-4297-49f5-995a-1ca5ba800724" (UID: "c5aca7fb-4297-49f5-995a-1ca5ba800724"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:47:07.290960 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.290950 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c5aca7fb-4297-49f5-995a-1ca5ba800724" (UID: "c5aca7fb-4297-49f5-995a-1ca5ba800724"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:47:07.291098 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.291029 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5aca7fb-4297-49f5-995a-1ca5ba800724-kube-api-access-cf7rp" (OuterVolumeSpecName: "kube-api-access-cf7rp") pod "c5aca7fb-4297-49f5-995a-1ca5ba800724" (UID: "c5aca7fb-4297-49f5-995a-1ca5ba800724"). InnerVolumeSpecName "kube-api-access-cf7rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:47:07.389562 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.389526 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-serving-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:07.389562 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.389559 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-trusted-ca-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:07.389562 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.389569 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-oauth-config\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:07.389823 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.389578 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:07.389823 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.389588 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-oauth-serving-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:07.389823 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.389597 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5aca7fb-4297-49f5-995a-1ca5ba800724-console-config\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:07.389823 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.389606 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf7rp\" (UniqueName: \"kubernetes.io/projected/c5aca7fb-4297-49f5-995a-1ca5ba800724-kube-api-access-cf7rp\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:07.959867 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.959832 2578 generic.go:356] "Generic (PLEG): container finished" podID="c5aca7fb-4297-49f5-995a-1ca5ba800724" containerID="1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3" exitCode=0 Sep 05 19:47:07.960038 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.959888 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556645967-fx9jh" Sep 05 19:47:07.960038 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.959922 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556645967-fx9jh" event={"ID":"c5aca7fb-4297-49f5-995a-1ca5ba800724","Type":"ContainerDied","Data":"1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3"} Sep 05 19:47:07.960038 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.960002 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556645967-fx9jh" event={"ID":"c5aca7fb-4297-49f5-995a-1ca5ba800724","Type":"ContainerDied","Data":"b9d90257e097bf205ff20965ee44f0fd0c879cf2fe385cdb6ed27a6e224a4e7d"} Sep 05 19:47:07.960038 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.960024 2578 scope.go:117] "RemoveContainer" containerID="1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3" Sep 05 19:47:07.968507 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.968490 2578 scope.go:117] "RemoveContainer" containerID="1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3" Sep 05 19:47:07.968774 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:47:07.968750 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3\": container with ID starting with 1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3 not found: ID does not exist" containerID="1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3" Sep 05 19:47:07.968841 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.968782 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3"} err="failed to get container status \"1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3\": rpc error: code = NotFound desc = could not find container \"1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3\": container with ID starting with 1442f79085570258a0d32682de9475257ab9341ef5abb7dce05bbc20ce300ef3 not found: ID does not exist" Sep 05 19:47:07.982684 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.982658 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556645967-fx9jh"] Sep 05 19:47:07.986275 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:07.986254 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-556645967-fx9jh"] Sep 05 19:47:08.011043 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:08.011009 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5aca7fb-4297-49f5-995a-1ca5ba800724" path="/var/lib/kubelet/pods/c5aca7fb-4297-49f5-995a-1ca5ba800724/volumes" Sep 05 19:47:09.957510 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:09.957479 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:47:09.961353 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:09.961330 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7cbf6b79f5-cqqhr" Sep 05 19:47:10.921498 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:10.921460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:47:10.924145 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:10.924110 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2aaad4-dc65-4f15-be9a-9fe0babba5b7-metrics-certs\") pod \"network-metrics-daemon-89gvd\" (UID: \"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7\") " pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:47:11.010834 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:11.010786 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-srspc\"" Sep 05 19:47:11.018396 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:11.018361 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89gvd" Sep 05 19:47:11.337510 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:11.337477 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-89gvd"] Sep 05 19:47:11.340729 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:47:11.340700 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2aaad4_dc65_4f15_be9a_9fe0babba5b7.slice/crio-29c4838461f34502c89e0917cf230ece796c33f59f5005b0abd2cb0ce9861522 WatchSource:0}: Error finding container 29c4838461f34502c89e0917cf230ece796c33f59f5005b0abd2cb0ce9861522: Status 404 returned error can't find the container with id 29c4838461f34502c89e0917cf230ece796c33f59f5005b0abd2cb0ce9861522 Sep 05 19:47:11.987902 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:11.987412 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-89gvd" event={"ID":"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7","Type":"ContainerStarted","Data":"29c4838461f34502c89e0917cf230ece796c33f59f5005b0abd2cb0ce9861522"} Sep 05 19:47:12.993096 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:12.993015 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-89gvd" event={"ID":"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7","Type":"ContainerStarted","Data":"010d0c373dfb1839e462996564467c2372db7396d61ec6bcfc21721ef800eb3d"} Sep 05 19:47:13.999572 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:13.999533 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-89gvd" event={"ID":"fe2aaad4-dc65-4f15-be9a-9fe0babba5b7","Type":"ContainerStarted","Data":"71481eb45e8337d1f5cf4f82ad185f30e5b1215e4faa1cef8e346fcdca8398c7"} Sep 05 19:47:14.017322 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:14.017252 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-89gvd" podStartSLOduration=252.556796918 podStartE2EDuration="4m14.017234415s" totalImagesPullingTime="1.460437497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:47:14.01673773 +0000 UTC m=+254.502038602" watchObservedRunningTime="2026-09-05 19:47:14.017234415 +0000 UTC m=+254.502535289" Sep 05 19:47:31.960724 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:31.960625 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:31.980673 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:31.980645 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:32.076640 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:32.076607 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:39.582359 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:47:39.582314 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rzzf9" podUID="99e4b881-669e-4df5-9feb-e59d740252bf" Sep 05 19:47:39.582359 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:47:39.582322 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" podUID="f57d845b-1d2a-4c86-abf5-2f72a04b4532" Sep 05 19:47:40.088542 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:40.088511 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rzzf9" Sep 05 19:47:40.088542 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:40.088541 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:47:43.323284 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:43.323238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:47:43.323760 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:43.323319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:47:43.328824 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:43.326442 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e4b881-669e-4df5-9feb-e59d740252bf-metrics-tls\") pod \"dns-default-rzzf9\" (UID: \"99e4b881-669e-4df5-9feb-e59d740252bf\") " pod="openshift-dns/dns-default-rzzf9" Sep 05 19:47:43.329655 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:43.329627 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f57d845b-1d2a-4c86-abf5-2f72a04b4532-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-rm5lj\" (UID: \"f57d845b-1d2a-4c86-abf5-2f72a04b4532\") " pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:47:43.392998 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:43.392959 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-64jqc\"" Sep 05 19:47:43.393762 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:43.393744 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6x9hn\"" Sep 05 19:47:43.400094 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:43.400068 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" Sep 05 19:47:43.400186 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:43.400095 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rzzf9" Sep 05 19:47:43.693951 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:43.693924 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-rm5lj"] Sep 05 19:47:43.695317 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:43.695292 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rzzf9"] Sep 05 19:47:43.696819 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:47:43.696776 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57d845b_1d2a_4c86_abf5_2f72a04b4532.slice/crio-813a9f82a481367395a90573054345f403457c2bcab375b9c56cf3abd3f19733 WatchSource:0}: Error finding container 813a9f82a481367395a90573054345f403457c2bcab375b9c56cf3abd3f19733: Status 404 returned error can't find the container with id 813a9f82a481367395a90573054345f403457c2bcab375b9c56cf3abd3f19733 Sep 05 19:47:43.697553 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:47:43.697536 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e4b881_669e_4df5_9feb_e59d740252bf.slice/crio-87d46fdb375d40d214eae039e4a0cfd9837ed54d8dab40136472c6be69068305 WatchSource:0}: Error finding container 87d46fdb375d40d214eae039e4a0cfd9837ed54d8dab40136472c6be69068305: Status 404 returned error can't find the container with id 87d46fdb375d40d214eae039e4a0cfd9837ed54d8dab40136472c6be69068305 Sep 05 19:47:44.103361 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:44.103322 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rzzf9" event={"ID":"99e4b881-669e-4df5-9feb-e59d740252bf","Type":"ContainerStarted","Data":"87d46fdb375d40d214eae039e4a0cfd9837ed54d8dab40136472c6be69068305"} Sep 05 19:47:44.104464 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:44.104439 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" event={"ID":"f57d845b-1d2a-4c86-abf5-2f72a04b4532","Type":"ContainerStarted","Data":"813a9f82a481367395a90573054345f403457c2bcab375b9c56cf3abd3f19733"} Sep 05 19:47:46.115107 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:46.115066 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rzzf9" event={"ID":"99e4b881-669e-4df5-9feb-e59d740252bf","Type":"ContainerStarted","Data":"635c7be2e88ccc1d730ce60f0667817fcd69a61c2875d2772e06da498cf98acd"} Sep 05 19:47:46.115632 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:46.115132 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rzzf9" event={"ID":"99e4b881-669e-4df5-9feb-e59d740252bf","Type":"ContainerStarted","Data":"b9762211ad41300720421e68cc577556beb42217fa4985f373972d82f23ee6aa"} Sep 05 19:47:46.115632 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:46.115189 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rzzf9" Sep 05 19:47:46.116632 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:46.116610 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" event={"ID":"f57d845b-1d2a-4c86-abf5-2f72a04b4532","Type":"ContainerStarted","Data":"93823948dd1828dba029e47654634835fdf049e0e805dc7324f4b059cb9eab34"} Sep 05 19:47:46.137811 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:46.137752 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rzzf9" podStartSLOduration=251.347598689 podStartE2EDuration="4m13.137733618s" totalImagesPullingTime="1.790134929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:47:46.136456616 +0000 UTC m=+286.621757501" watchObservedRunningTime="2026-09-05 19:47:46.137733618 +0000 UTC m=+286.623034491" Sep 05 19:47:46.156493 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:46.156432 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-rm5lj" podStartSLOduration=262.370980614 podStartE2EDuration="4m24.156417992s" totalImagesPullingTime="1.785437378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:47:46.155486076 +0000 UTC m=+286.640786949" watchObservedRunningTime="2026-09-05 19:47:46.156417992 +0000 UTC m=+286.641718864" Sep 05 19:47:50.006113 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.006072 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 05 19:47:50.008151 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.007893 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="prometheus" containerID="cri-o://357081889e3e02d9977fbc1a3b02e53d51fcbda463e71577d5849f98860d5fae" gracePeriod=600 Sep 05 19:47:50.008151 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.007936 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="thanos-sidecar" containerID="cri-o://d0999d74a64d79a88c602eb47bfe48981d55d6365ad61df97cca5ef9e10401a4" gracePeriod=600 Sep 05 19:47:50.008151 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.007968 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="config-reloader" containerID="cri-o://1a082482174d56b19f7b31f26d69a762c084458448695d9735b2e4b35509c14c" gracePeriod=600 Sep 05 19:47:50.008151 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.007897 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="kube-rbac-proxy" containerID="cri-o://b37fe8bcef27d7b67c9cef84e4d71fad3d9cdf4f3a4ae20b86c9c84cd7fac7b2" gracePeriod=600 Sep 05 19:47:50.008151 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.007951 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://08eccb3c53e839c0bafefefd74f51f9811951221aa9f13cf57df0ca238047075" gracePeriod=600 Sep 05 19:47:50.008151 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.008116 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="kube-rbac-proxy-web" containerID="cri-o://54adaefad14811f007a1528c7c55589e97aa9f2728d18224450faa467ad58410" gracePeriod=600 Sep 05 19:47:50.150712 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.150672 2578 generic.go:356] "Generic (PLEG): container finished" podID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerID="08eccb3c53e839c0bafefefd74f51f9811951221aa9f13cf57df0ca238047075" exitCode=0 Sep 05 19:47:50.150712 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.150699 2578 generic.go:356] "Generic (PLEG): container finished" podID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerID="b37fe8bcef27d7b67c9cef84e4d71fad3d9cdf4f3a4ae20b86c9c84cd7fac7b2" exitCode=0 Sep 05 19:47:50.150712 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.150707 2578 generic.go:356] "Generic (PLEG): container finished" podID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerID="54adaefad14811f007a1528c7c55589e97aa9f2728d18224450faa467ad58410" exitCode=0 Sep 05 19:47:50.150712 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.150712 2578 generic.go:356] "Generic (PLEG): container finished" podID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerID="d0999d74a64d79a88c602eb47bfe48981d55d6365ad61df97cca5ef9e10401a4" exitCode=0 Sep 05 19:47:50.150712 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.150720 2578 generic.go:356] "Generic (PLEG): container finished" podID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerID="1a082482174d56b19f7b31f26d69a762c084458448695d9735b2e4b35509c14c" exitCode=0 Sep 05 19:47:50.150712 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.150724 2578 generic.go:356] "Generic (PLEG): container finished" podID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerID="357081889e3e02d9977fbc1a3b02e53d51fcbda463e71577d5849f98860d5fae" exitCode=0 Sep 05 19:47:50.151225 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.150740 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerDied","Data":"08eccb3c53e839c0bafefefd74f51f9811951221aa9f13cf57df0ca238047075"} Sep 05 19:47:50.151225 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.150821 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerDied","Data":"b37fe8bcef27d7b67c9cef84e4d71fad3d9cdf4f3a4ae20b86c9c84cd7fac7b2"} Sep 05 19:47:50.151225 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.150849 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerDied","Data":"54adaefad14811f007a1528c7c55589e97aa9f2728d18224450faa467ad58410"} Sep 05 19:47:50.151225 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.150875 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerDied","Data":"d0999d74a64d79a88c602eb47bfe48981d55d6365ad61df97cca5ef9e10401a4"} Sep 05 19:47:50.151225 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.150898 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerDied","Data":"1a082482174d56b19f7b31f26d69a762c084458448695d9735b2e4b35509c14c"} Sep 05 19:47:50.151225 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.150921 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerDied","Data":"357081889e3e02d9977fbc1a3b02e53d51fcbda463e71577d5849f98860d5fae"} Sep 05 19:47:50.261535 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.261475 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:50.382540 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.382506 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-serving-certs-ca-bundle\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.382715 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.382549 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-db\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.382715 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.382570 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef17468b-7f9b-449c-a708-bed11f271a8e-config-out\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.382715 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.382589 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef17468b-7f9b-449c-a708-bed11f271a8e-tls-assets\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.382715 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.382606 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.382715 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.382641 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-0\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.382715 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.382663 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-kubelet-serving-ca-bundle\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.382715 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.382681 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-1\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.382715 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.382702 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.383156 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.382726 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-metrics-client-certs\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.383156 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.382996 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:47:50.383156 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383105 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:47:50.383307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383162 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-tls\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.383307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383196 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z45r\" (UniqueName: \"kubernetes.io/projected/ef17468b-7f9b-449c-a708-bed11f271a8e-kube-api-access-6z45r\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.383307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383223 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-config\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.383307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383264 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-thanos-prometheus-http-client-file\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.383506 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383316 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-web-config\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.383506 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383344 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-metrics-client-ca\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.383506 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383353 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:47:50.383506 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383388 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-2\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.383506 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383414 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-trusted-ca-bundle\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.383506 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383450 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-grpc-tls\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.383506 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383488 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-kube-rbac-proxy\") pod \"ef17468b-7f9b-449c-a708-bed11f271a8e\" (UID: \"ef17468b-7f9b-449c-a708-bed11f271a8e\") " Sep 05 19:47:50.383897 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383728 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.383897 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383746 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.383897 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383760 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.383897 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383866 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:47:50.384101 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.383901 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 19:47:50.384219 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.384191 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:47:50.384972 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.384701 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:47:50.385087 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.385012 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:47:50.388364 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.386335 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:47:50.388364 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.386721 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:47:50.388364 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.387609 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef17468b-7f9b-449c-a708-bed11f271a8e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:47:50.388364 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.387731 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:47:50.388364 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.387855 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:47:50.388689 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.388480 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-config" (OuterVolumeSpecName: "config") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:47:50.388689 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.388658 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef17468b-7f9b-449c-a708-bed11f271a8e-config-out" (OuterVolumeSpecName: "config-out") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 19:47:50.389218 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.389191 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:47:50.389384 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.389358 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef17468b-7f9b-449c-a708-bed11f271a8e-kube-api-access-6z45r" (OuterVolumeSpecName: "kube-api-access-6z45r") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "kube-api-access-6z45r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:47:50.390135 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.390101 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:47:50.390282 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.390263 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:47:50.398403 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.398373 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-web-config" (OuterVolumeSpecName: "web-config") pod "ef17468b-7f9b-449c-a708-bed11f271a8e" (UID: "ef17468b-7f9b-449c-a708-bed11f271a8e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:47:50.484914 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.484866 2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-config\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.484914 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.484912 2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.484914 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.484924 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-web-config\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.484935 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-configmap-metrics-client-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.484944 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.484953 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.484962 2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-grpc-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.484970 2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-kube-rbac-proxy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.484978 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-db\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.484986 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef17468b-7f9b-449c-a708-bed11f271a8e-config-out\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.484995 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef17468b-7f9b-449c-a708-bed11f271a8e-tls-assets\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.485003 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.485012 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef17468b-7f9b-449c-a708-bed11f271a8e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.485022 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.485031 2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-metrics-client-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.485039 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef17468b-7f9b-449c-a708-bed11f271a8e-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:50.485112 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:50.485047 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z45r\" (UniqueName: \"kubernetes.io/projected/ef17468b-7f9b-449c-a708-bed11f271a8e-kube-api-access-6z45r\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:47:51.158597 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.158561 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef17468b-7f9b-449c-a708-bed11f271a8e","Type":"ContainerDied","Data":"006eb5f822d0db1499639759a18f8baa41af2213eb54e609c9bd469b8ed8e24b"} Sep 05 19:47:51.159063 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.158640 2578 scope.go:117] "RemoveContainer" containerID="08eccb3c53e839c0bafefefd74f51f9811951221aa9f13cf57df0ca238047075" Sep 05 19:47:51.159063 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.158637 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.166756 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.166732 2578 scope.go:117] "RemoveContainer" containerID="b37fe8bcef27d7b67c9cef84e4d71fad3d9cdf4f3a4ae20b86c9c84cd7fac7b2" Sep 05 19:47:51.173959 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.173940 2578 scope.go:117] "RemoveContainer" containerID="54adaefad14811f007a1528c7c55589e97aa9f2728d18224450faa467ad58410" Sep 05 19:47:51.180612 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.180591 2578 scope.go:117] "RemoveContainer" containerID="d0999d74a64d79a88c602eb47bfe48981d55d6365ad61df97cca5ef9e10401a4" Sep 05 19:47:51.186993 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.186967 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 05 19:47:51.188931 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.188915 2578 scope.go:117] "RemoveContainer" containerID="1a082482174d56b19f7b31f26d69a762c084458448695d9735b2e4b35509c14c" Sep 05 19:47:51.197470 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.197443 2578 scope.go:117] "RemoveContainer" containerID="357081889e3e02d9977fbc1a3b02e53d51fcbda463e71577d5849f98860d5fae" Sep 05 19:47:51.198410 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.198386 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 05 19:47:51.205438 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.205419 2578 scope.go:117] "RemoveContainer" containerID="1782e9465eec1edeb32bfc41ce3d06849964cbe9a8dd61d4294c2928185a4709" Sep 05 19:47:51.234696 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.234667 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 05 19:47:51.235001 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.234985 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="prometheus" Sep 05 19:47:51.235091 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235004 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="prometheus" Sep 05 19:47:51.235091 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235021 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5aca7fb-4297-49f5-995a-1ca5ba800724" containerName="console" Sep 05 19:47:51.235091 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235029 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5aca7fb-4297-49f5-995a-1ca5ba800724" containerName="console" Sep 05 19:47:51.235091 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235049 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="kube-rbac-proxy-thanos" Sep 05 19:47:51.235091 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235059 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="kube-rbac-proxy-thanos" Sep 05 19:47:51.235091 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235072 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="init-config-reloader" Sep 05 19:47:51.235091 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235081 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="init-config-reloader" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235100 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de746a5-fc32-4c55-97d4-cb55fe39360b" containerName="console" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235108 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de746a5-fc32-4c55-97d4-cb55fe39360b" containerName="console" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235119 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="config-reloader" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235129 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="config-reloader" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235138 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="thanos-sidecar" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235146 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="thanos-sidecar" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235160 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="kube-rbac-proxy-web" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235169 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="kube-rbac-proxy-web" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235183 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="kube-rbac-proxy" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235191 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="kube-rbac-proxy" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235258 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="kube-rbac-proxy-web" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235270 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="config-reloader" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235279 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="kube-rbac-proxy" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235290 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="prometheus" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235298 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5aca7fb-4297-49f5-995a-1ca5ba800724" containerName="console" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235305 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="thanos-sidecar" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235313 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" containerName="kube-rbac-proxy-thanos" Sep 05 19:47:51.235414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.235322 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de746a5-fc32-4c55-97d4-cb55fe39360b" containerName="console" Sep 05 19:47:51.239341 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.239320 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.242457 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.242438 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9pwdw\"" Sep 05 19:47:51.245007 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.244978 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 05 19:47:51.245121 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.245058 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 05 19:47:51.245266 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.245247 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 05 19:47:51.245564 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.245280 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 05 19:47:51.245564 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.245315 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 05 19:47:51.245564 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.245357 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 05 19:47:51.245564 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.245370 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 05 19:47:51.245564 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.245249 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 05 19:47:51.245564 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.245464 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 05 19:47:51.245564 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.245515 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 05 19:47:51.245564 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.245515 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 05 19:47:51.245564 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.245559 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 05 19:47:51.246123 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.245925 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3hafphn87bks3\"" Sep 05 19:47:51.254261 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.254230 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 05 19:47:51.256598 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.256570 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 05 19:47:51.262738 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.262706 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 05 19:47:51.290956 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.290858 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.290956 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.290900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.290956 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.290921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.290956 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.290949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-config\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291295 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291024 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c46619e-0f25-424d-9d30-b6e50d11bb14-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291295 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291066 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291295 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291295 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl6t4\" (UniqueName: \"kubernetes.io/projected/9c46619e-0f25-424d-9d30-b6e50d11bb14-kube-api-access-jl6t4\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291295 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291198 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291295 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291295 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291253 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c46619e-0f25-424d-9d30-b6e50d11bb14-config-out\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291295 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291655 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-web-config\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291655 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291655 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291655 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291415 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291655 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291655 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291655 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.291655 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.291556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.392948 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.392911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c46619e-0f25-424d-9d30-b6e50d11bb14-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.392948 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.392960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.393190 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.392999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.393190 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.393030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl6t4\" (UniqueName: \"kubernetes.io/projected/9c46619e-0f25-424d-9d30-b6e50d11bb14-kube-api-access-jl6t4\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.393190 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.393059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.393190 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.393085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.393190 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.393107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c46619e-0f25-424d-9d30-b6e50d11bb14-config-out\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.393522 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.393493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.393605 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.393552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-web-config\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.393858 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.393830 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.393991 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.393970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.394057 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.394007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.394057 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.394042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.394057 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.394042 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.394239 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.394066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.394239 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.394091 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.394239 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.394135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.394239 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.394167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.394239 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.394199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.394239 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.394221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.394545 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.394245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.394545 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.394276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-config\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.395012 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.394985 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.395748 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.395629 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.396238 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.395949 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.396339 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.396248 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c46619e-0f25-424d-9d30-b6e50d11bb14-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.396339 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.396271 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-web-config\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.396665 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.396638 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.396980 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.396959 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-config\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.397108 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.397013 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.397322 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.397294 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.397744 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.397717 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c46619e-0f25-424d-9d30-b6e50d11bb14-config-out\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.397880 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.397855 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46619e-0f25-424d-9d30-b6e50d11bb14-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.398302 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.398277 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.398537 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.398516 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.398965 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.398944 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.399009 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.398968 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.399353 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.399338 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.399771 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.399755 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9c46619e-0f25-424d-9d30-b6e50d11bb14-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.403707 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.403683 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl6t4\" (UniqueName: \"kubernetes.io/projected/9c46619e-0f25-424d-9d30-b6e50d11bb14-kube-api-access-jl6t4\") pod \"prometheus-k8s-0\" (UID: \"9c46619e-0f25-424d-9d30-b6e50d11bb14\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.554088 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.553992 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:51.838923 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:51.838896 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 05 19:47:51.841171 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:47:51.841142 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c46619e_0f25_424d_9d30_b6e50d11bb14.slice/crio-d31945286a510e2576df63bfd4b3cb450470ed8115f5e8a05eabbd35874a4af7 WatchSource:0}: Error finding container d31945286a510e2576df63bfd4b3cb450470ed8115f5e8a05eabbd35874a4af7: Status 404 returned error can't find the container with id d31945286a510e2576df63bfd4b3cb450470ed8115f5e8a05eabbd35874a4af7 Sep 05 19:47:52.011291 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:52.011256 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef17468b-7f9b-449c-a708-bed11f271a8e" path="/var/lib/kubelet/pods/ef17468b-7f9b-449c-a708-bed11f271a8e/volumes" Sep 05 19:47:52.164318 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:52.164235 2578 generic.go:356] "Generic (PLEG): container finished" podID="9c46619e-0f25-424d-9d30-b6e50d11bb14" containerID="0a81a7b8cc4b18a25dc859c9f215eec0624e850582465e04047a25e8104c901a" exitCode=0 Sep 05 19:47:52.164749 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:52.164320 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c46619e-0f25-424d-9d30-b6e50d11bb14","Type":"ContainerDied","Data":"0a81a7b8cc4b18a25dc859c9f215eec0624e850582465e04047a25e8104c901a"} Sep 05 19:47:52.164749 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:52.164393 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c46619e-0f25-424d-9d30-b6e50d11bb14","Type":"ContainerStarted","Data":"d31945286a510e2576df63bfd4b3cb450470ed8115f5e8a05eabbd35874a4af7"} Sep 05 19:47:53.179159 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:53.179121 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c46619e-0f25-424d-9d30-b6e50d11bb14","Type":"ContainerStarted","Data":"78d012fa519fef52b447e6005b521c4fd0e02d07ae5d8b862bae64a7822b253c"} Sep 05 19:47:53.179646 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:53.179227 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c46619e-0f25-424d-9d30-b6e50d11bb14","Type":"ContainerStarted","Data":"db1085ac0a4eae7529691030e3fd4b97bff5aa2cbf67ad8696af7792efb74b1b"} Sep 05 19:47:53.179646 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:53.179277 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c46619e-0f25-424d-9d30-b6e50d11bb14","Type":"ContainerStarted","Data":"55afa24563f0781a809b3938135e806efd0cf2eeeb51347ffc8bd63258bde308"} Sep 05 19:47:53.179646 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:53.179322 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c46619e-0f25-424d-9d30-b6e50d11bb14","Type":"ContainerStarted","Data":"9b8050eb07b076d6f62e3ec0271999465612db56eb7f68ade7277115dee65a5f"} Sep 05 19:47:53.179646 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:53.179365 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c46619e-0f25-424d-9d30-b6e50d11bb14","Type":"ContainerStarted","Data":"0fb956fb4403b6965f842f4fd12c5f3edebf1889b9bf9241a9a56f28d005e11f"} Sep 05 19:47:53.179646 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:53.179405 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c46619e-0f25-424d-9d30-b6e50d11bb14","Type":"ContainerStarted","Data":"d5bfb44ab10c9a630fe7ec4a142c92a6684858311e4a9f79226b03219be137bd"} Sep 05 19:47:53.216981 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:53.216916 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.196213229 podStartE2EDuration="2.216892409s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.67918ms" isStatefulPod=false podCreationTimestamp="2026-09-05 19:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:47:53.21419254 +0000 UTC m=+293.699493415" watchObservedRunningTime="2026-09-05 19:47:53.216892409 +0000 UTC m=+293.702193283" Sep 05 19:47:56.137461 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:56.137431 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rzzf9" Sep 05 19:47:56.554865 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:56.554828 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:47:59.938676 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:59.938652 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 19:47:59.939091 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:59.938925 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 19:47:59.942249 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:47:59.942230 2578 kubelet.go:1648] "Image garbage collection succeeded" Sep 05 19:48:51.554502 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:48:51.554463 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:48:51.570011 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:48:51.569985 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:48:52.411772 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:48:52.411702 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 05 19:50:41.739598 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:41.739560 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nm97p"] Sep 05 19:50:41.743306 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:41.743284 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" Sep 05 19:50:41.746234 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:41.746212 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 05 19:50:41.747252 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:41.747231 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rhw9b\"" Sep 05 19:50:41.747413 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:41.747235 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 05 19:50:41.750195 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:41.750173 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nm97p"] Sep 05 19:50:41.767313 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:41.767283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bzw6\" (UniqueName: \"kubernetes.io/projected/659b2376-46af-46ea-8065-1771ce9ea740-kube-api-access-8bzw6\") pod \"cert-manager-webhook-6486ffd997-nm97p\" (UID: \"659b2376-46af-46ea-8065-1771ce9ea740\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" Sep 05 19:50:41.767485 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:41.767344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/659b2376-46af-46ea-8065-1771ce9ea740-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nm97p\" (UID: \"659b2376-46af-46ea-8065-1771ce9ea740\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" Sep 05 19:50:41.868436 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:41.868396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bzw6\" (UniqueName: \"kubernetes.io/projected/659b2376-46af-46ea-8065-1771ce9ea740-kube-api-access-8bzw6\") pod \"cert-manager-webhook-6486ffd997-nm97p\" (UID: \"659b2376-46af-46ea-8065-1771ce9ea740\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" Sep 05 19:50:41.868639 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:41.868455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/659b2376-46af-46ea-8065-1771ce9ea740-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nm97p\" (UID: \"659b2376-46af-46ea-8065-1771ce9ea740\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" Sep 05 19:50:41.878285 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:41.878245 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/659b2376-46af-46ea-8065-1771ce9ea740-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nm97p\" (UID: \"659b2376-46af-46ea-8065-1771ce9ea740\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" Sep 05 19:50:41.878414 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:41.878295 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bzw6\" (UniqueName: \"kubernetes.io/projected/659b2376-46af-46ea-8065-1771ce9ea740-kube-api-access-8bzw6\") pod \"cert-manager-webhook-6486ffd997-nm97p\" (UID: \"659b2376-46af-46ea-8065-1771ce9ea740\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" Sep 05 19:50:42.061746 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:42.061707 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" Sep 05 19:50:42.333527 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:42.333451 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nm97p"] Sep 05 19:50:42.336948 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:50:42.336913 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod659b2376_46af_46ea_8065_1771ce9ea740.slice/crio-59cc67d97c6d1e01b99bff1a67dbd78b32671cdb9b90de2e5f20b77e3ed05bca WatchSource:0}: Error finding container 59cc67d97c6d1e01b99bff1a67dbd78b32671cdb9b90de2e5f20b77e3ed05bca: Status 404 returned error can't find the container with id 59cc67d97c6d1e01b99bff1a67dbd78b32671cdb9b90de2e5f20b77e3ed05bca Sep 05 19:50:42.338851 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:42.338834 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 19:50:42.800496 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:42.800460 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" event={"ID":"659b2376-46af-46ea-8065-1771ce9ea740","Type":"ContainerStarted","Data":"59cc67d97c6d1e01b99bff1a67dbd78b32671cdb9b90de2e5f20b77e3ed05bca"} Sep 05 19:50:46.818891 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:46.818784 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" event={"ID":"659b2376-46af-46ea-8065-1771ce9ea740","Type":"ContainerStarted","Data":"af9ea61248293fb4a5d5c2a106ea3f5b9dacc752bd8b2b42f16af93aef5475b1"} Sep 05 19:50:46.819264 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:46.818909 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" Sep 05 19:50:46.838323 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:46.838268 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" podStartSLOduration=1.6286595209999999 podStartE2EDuration="5.83825271s" totalImagesPullingTime="4.209593189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:50:46.835971488 +0000 UTC m=+467.321272361" watchObservedRunningTime="2026-09-05 19:50:46.83825271 +0000 UTC m=+467.323553583" Sep 05 19:50:52.774617 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.774576 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb"] Sep 05 19:50:52.777839 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.777813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb" Sep 05 19:50:52.780883 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.780863 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 05 19:50:52.780883 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.780868 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 05 19:50:52.782026 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.782010 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-kv5tj\"" Sep 05 19:50:52.785675 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.785651 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb"] Sep 05 19:50:52.825577 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.825543 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nm97p" Sep 05 19:50:52.854688 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.854658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/286a579d-14bd-4fef-a378-8dbef9fb1d7d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8q5nb\" (UID: \"286a579d-14bd-4fef-a378-8dbef9fb1d7d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb" Sep 05 19:50:52.854862 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.854704 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mlcz\" (UniqueName: \"kubernetes.io/projected/286a579d-14bd-4fef-a378-8dbef9fb1d7d-kube-api-access-9mlcz\") pod \"openshift-lws-operator-fd8ccff4c-8q5nb\" (UID: \"286a579d-14bd-4fef-a378-8dbef9fb1d7d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb" Sep 05 19:50:52.956030 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.955989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/286a579d-14bd-4fef-a378-8dbef9fb1d7d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8q5nb\" (UID: \"286a579d-14bd-4fef-a378-8dbef9fb1d7d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb" Sep 05 19:50:52.956030 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.956035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mlcz\" (UniqueName: \"kubernetes.io/projected/286a579d-14bd-4fef-a378-8dbef9fb1d7d-kube-api-access-9mlcz\") pod \"openshift-lws-operator-fd8ccff4c-8q5nb\" (UID: \"286a579d-14bd-4fef-a378-8dbef9fb1d7d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb" Sep 05 19:50:52.956356 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.956334 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/286a579d-14bd-4fef-a378-8dbef9fb1d7d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8q5nb\" (UID: \"286a579d-14bd-4fef-a378-8dbef9fb1d7d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb" Sep 05 19:50:52.966288 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:52.966248 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mlcz\" (UniqueName: \"kubernetes.io/projected/286a579d-14bd-4fef-a378-8dbef9fb1d7d-kube-api-access-9mlcz\") pod \"openshift-lws-operator-fd8ccff4c-8q5nb\" (UID: \"286a579d-14bd-4fef-a378-8dbef9fb1d7d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb" Sep 05 19:50:53.087164 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:53.087061 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb" Sep 05 19:50:53.356088 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:53.356010 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb"] Sep 05 19:50:53.359035 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:50:53.359008 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod286a579d_14bd_4fef_a378_8dbef9fb1d7d.slice/crio-c986f4948195809ad69a38f1c766288e97387474e02d4cf76705f04f481cb270 WatchSource:0}: Error finding container c986f4948195809ad69a38f1c766288e97387474e02d4cf76705f04f481cb270: Status 404 returned error can't find the container with id c986f4948195809ad69a38f1c766288e97387474e02d4cf76705f04f481cb270 Sep 05 19:50:53.847852 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:53.847813 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb" event={"ID":"286a579d-14bd-4fef-a378-8dbef9fb1d7d","Type":"ContainerStarted","Data":"c986f4948195809ad69a38f1c766288e97387474e02d4cf76705f04f481cb270"} Sep 05 19:50:56.862933 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:56.862893 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb" event={"ID":"286a579d-14bd-4fef-a378-8dbef9fb1d7d","Type":"ContainerStarted","Data":"948c23dbe8b00695f4ce96e45b521eb3147e975b47d1256f270de87a30aaf1d9"} Sep 05 19:50:56.882297 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:50:56.882244 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8q5nb" podStartSLOduration=2.3746216430000002 podStartE2EDuration="4.882225502s" totalImagesPullingTime="2.507603859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:50:56.88150418 +0000 UTC m=+477.366805045" watchObservedRunningTime="2026-09-05 19:50:56.882225502 +0000 UTC m=+477.367526375" Sep 05 19:51:11.983819 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:11.983775 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g"] Sep 05 19:51:11.990068 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:11.990040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:11.993330 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:11.993309 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 05 19:51:11.993486 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:11.993461 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ngx5g\"" Sep 05 19:51:11.993658 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:11.993603 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 05 19:51:11.993950 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:11.993935 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 05 19:51:11.997183 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:11.997166 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 05 19:51:12.001935 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.001909 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g"] Sep 05 19:51:12.118236 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.118192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d49fb878-4312-4d3c-92d7-f0654de8071d-tmp\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.118421 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.118254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d49fb878-4312-4d3c-92d7-f0654de8071d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.118421 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.118332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8gzr\" (UniqueName: \"kubernetes.io/projected/d49fb878-4312-4d3c-92d7-f0654de8071d-kube-api-access-n8gzr\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.118421 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.118367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d49fb878-4312-4d3c-92d7-f0654de8071d-webhook-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.118538 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.118443 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d49fb878-4312-4d3c-92d7-f0654de8071d-manifests\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.219708 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.219657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d49fb878-4312-4d3c-92d7-f0654de8071d-manifests\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.219708 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.219713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d49fb878-4312-4d3c-92d7-f0654de8071d-tmp\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.219958 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.219747 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d49fb878-4312-4d3c-92d7-f0654de8071d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.219958 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.219774 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gzr\" (UniqueName: \"kubernetes.io/projected/d49fb878-4312-4d3c-92d7-f0654de8071d-kube-api-access-n8gzr\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.219958 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.219792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d49fb878-4312-4d3c-92d7-f0654de8071d-webhook-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.220155 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.220102 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d49fb878-4312-4d3c-92d7-f0654de8071d-manifests\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.222144 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.222116 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d49fb878-4312-4d3c-92d7-f0654de8071d-tmp\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.222460 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.222439 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d49fb878-4312-4d3c-92d7-f0654de8071d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.222501 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.222439 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d49fb878-4312-4d3c-92d7-f0654de8071d-webhook-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.229600 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.229573 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gzr\" (UniqueName: \"kubernetes.io/projected/d49fb878-4312-4d3c-92d7-f0654de8071d-kube-api-access-n8gzr\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-kv67g\" (UID: \"d49fb878-4312-4d3c-92d7-f0654de8071d\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.299766 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.299732 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:12.577328 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.577251 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g"] Sep 05 19:51:12.580603 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:51:12.580576 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49fb878_4312_4d3c_92d7_f0654de8071d.slice/crio-33c0e6b9be4de68e645ec593ab0b8f7e429d19869ce045ddaa561d52d64a4237 WatchSource:0}: Error finding container 33c0e6b9be4de68e645ec593ab0b8f7e429d19869ce045ddaa561d52d64a4237: Status 404 returned error can't find the container with id 33c0e6b9be4de68e645ec593ab0b8f7e429d19869ce045ddaa561d52d64a4237 Sep 05 19:51:12.929019 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:12.928930 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" event={"ID":"d49fb878-4312-4d3c-92d7-f0654de8071d","Type":"ContainerStarted","Data":"33c0e6b9be4de68e645ec593ab0b8f7e429d19869ce045ddaa561d52d64a4237"} Sep 05 19:51:15.944046 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:15.943989 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" event={"ID":"d49fb878-4312-4d3c-92d7-f0654de8071d","Type":"ContainerStarted","Data":"d8977850c0870c31c041afa1611d2f4a8340a58f8452d526cf49f28cae1a6615"} Sep 05 19:51:17.955407 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:17.955374 2578 generic.go:356] "Generic (PLEG): container finished" podID="d49fb878-4312-4d3c-92d7-f0654de8071d" containerID="d8977850c0870c31c041afa1611d2f4a8340a58f8452d526cf49f28cae1a6615" exitCode=0 Sep 05 19:51:17.956000 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:17.955453 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" event={"ID":"d49fb878-4312-4d3c-92d7-f0654de8071d","Type":"ContainerDied","Data":"d8977850c0870c31c041afa1611d2f4a8340a58f8452d526cf49f28cae1a6615"} Sep 05 19:51:18.961316 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:18.961279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" event={"ID":"d49fb878-4312-4d3c-92d7-f0654de8071d","Type":"ContainerStarted","Data":"450d78abb81d0eb01d0ed7c3796469376fa1ae344a00c8369742234b2b0e12ee"} Sep 05 19:51:18.961720 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:18.961598 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:18.993242 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:18.993171 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" podStartSLOduration=3.343690811 podStartE2EDuration="7.99315454s" totalImagesPullingTime="3.553850237s" totalInitContainerRuntime="1.095613492s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:18.986835061 +0000 UTC m=+499.472135934" watchObservedRunningTime="2026-09-05 19:51:18.99315454 +0000 UTC m=+499.478455412" Sep 05 19:51:29.342538 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.342496 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7"] Sep 05 19:51:29.345755 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.345733 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.350061 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.350038 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 05 19:51:29.350155 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.350058 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 05 19:51:29.350155 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.350042 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 05 19:51:29.350155 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.350063 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-grxmh\"" Sep 05 19:51:29.357319 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.357297 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7"] Sep 05 19:51:29.464744 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.464707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae9ff93a-5782-4138-8d45-40ddc61ea16d-cert\") pod \"lws-controller-manager-6b9f469976-tscz7\" (UID: \"ae9ff93a-5782-4138-8d45-40ddc61ea16d\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.464744 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.464754 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae9ff93a-5782-4138-8d45-40ddc61ea16d-metrics-cert\") pod \"lws-controller-manager-6b9f469976-tscz7\" (UID: \"ae9ff93a-5782-4138-8d45-40ddc61ea16d\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.464989 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.464869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae9ff93a-5782-4138-8d45-40ddc61ea16d-manager-config\") pod \"lws-controller-manager-6b9f469976-tscz7\" (UID: \"ae9ff93a-5782-4138-8d45-40ddc61ea16d\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.464989 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.464928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4xmt\" (UniqueName: \"kubernetes.io/projected/ae9ff93a-5782-4138-8d45-40ddc61ea16d-kube-api-access-g4xmt\") pod \"lws-controller-manager-6b9f469976-tscz7\" (UID: \"ae9ff93a-5782-4138-8d45-40ddc61ea16d\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.566102 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.566061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae9ff93a-5782-4138-8d45-40ddc61ea16d-manager-config\") pod \"lws-controller-manager-6b9f469976-tscz7\" (UID: \"ae9ff93a-5782-4138-8d45-40ddc61ea16d\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.566102 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.566111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4xmt\" (UniqueName: \"kubernetes.io/projected/ae9ff93a-5782-4138-8d45-40ddc61ea16d-kube-api-access-g4xmt\") pod \"lws-controller-manager-6b9f469976-tscz7\" (UID: \"ae9ff93a-5782-4138-8d45-40ddc61ea16d\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.566307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.566224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae9ff93a-5782-4138-8d45-40ddc61ea16d-cert\") pod \"lws-controller-manager-6b9f469976-tscz7\" (UID: \"ae9ff93a-5782-4138-8d45-40ddc61ea16d\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.566307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.566268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae9ff93a-5782-4138-8d45-40ddc61ea16d-metrics-cert\") pod \"lws-controller-manager-6b9f469976-tscz7\" (UID: \"ae9ff93a-5782-4138-8d45-40ddc61ea16d\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.566737 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.566716 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae9ff93a-5782-4138-8d45-40ddc61ea16d-manager-config\") pod \"lws-controller-manager-6b9f469976-tscz7\" (UID: \"ae9ff93a-5782-4138-8d45-40ddc61ea16d\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.568904 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.568880 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae9ff93a-5782-4138-8d45-40ddc61ea16d-cert\") pod \"lws-controller-manager-6b9f469976-tscz7\" (UID: \"ae9ff93a-5782-4138-8d45-40ddc61ea16d\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.569008 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.568880 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae9ff93a-5782-4138-8d45-40ddc61ea16d-metrics-cert\") pod \"lws-controller-manager-6b9f469976-tscz7\" (UID: \"ae9ff93a-5782-4138-8d45-40ddc61ea16d\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.574174 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.574154 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4xmt\" (UniqueName: \"kubernetes.io/projected/ae9ff93a-5782-4138-8d45-40ddc61ea16d-kube-api-access-g4xmt\") pod \"lws-controller-manager-6b9f469976-tscz7\" (UID: \"ae9ff93a-5782-4138-8d45-40ddc61ea16d\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.655828 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.655723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:29.924885 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.924788 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7"] Sep 05 19:51:29.928336 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:51:29.928312 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9ff93a_5782_4138_8d45_40ddc61ea16d.slice/crio-8bb48557f79abe943a7d4f7ab46026de577d1e25be83ea229d5d251965939596 WatchSource:0}: Error finding container 8bb48557f79abe943a7d4f7ab46026de577d1e25be83ea229d5d251965939596: Status 404 returned error can't find the container with id 8bb48557f79abe943a7d4f7ab46026de577d1e25be83ea229d5d251965939596 Sep 05 19:51:29.968425 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:29.968397 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-kv67g" Sep 05 19:51:30.012133 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:30.012094 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" event={"ID":"ae9ff93a-5782-4138-8d45-40ddc61ea16d","Type":"ContainerStarted","Data":"8bb48557f79abe943a7d4f7ab46026de577d1e25be83ea229d5d251965939596"} Sep 05 19:51:32.019753 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.019707 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" event={"ID":"ae9ff93a-5782-4138-8d45-40ddc61ea16d","Type":"ContainerStarted","Data":"53b9bd3513fdd8b324a4feac00112d89c5513d53e7402c34c963783b15b7aa8f"} Sep 05 19:51:32.020162 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.019850 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:51:32.066879 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.066821 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" podStartSLOduration=1.508398105 podStartE2EDuration="3.066784464s" totalImagesPullingTime="1.558386359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:32.065915021 +0000 UTC m=+512.551215906" watchObservedRunningTime="2026-09-05 19:51:32.066784464 +0000 UTC m=+512.552085336" Sep 05 19:51:32.186376 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.186336 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv"] Sep 05 19:51:32.189838 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.189813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" Sep 05 19:51:32.192990 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.192963 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 05 19:51:32.193121 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.192967 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 05 19:51:32.193121 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.193000 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 05 19:51:32.193268 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.193254 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6r4cp\"" Sep 05 19:51:32.193371 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.193353 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 05 19:51:32.211558 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.211527 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv"] Sep 05 19:51:32.290960 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.290864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/617ea30d-e653-4a6f-882b-3ea090eee8ca-tls-certs\") pod \"kube-auth-proxy-84fbbb6876-9dwpv\" (UID: \"617ea30d-e653-4a6f-882b-3ea090eee8ca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" Sep 05 19:51:32.290960 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.290919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/617ea30d-e653-4a6f-882b-3ea090eee8ca-tmp\") pod \"kube-auth-proxy-84fbbb6876-9dwpv\" (UID: \"617ea30d-e653-4a6f-882b-3ea090eee8ca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" Sep 05 19:51:32.290960 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.290940 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f9tg\" (UniqueName: \"kubernetes.io/projected/617ea30d-e653-4a6f-882b-3ea090eee8ca-kube-api-access-9f9tg\") pod \"kube-auth-proxy-84fbbb6876-9dwpv\" (UID: \"617ea30d-e653-4a6f-882b-3ea090eee8ca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" Sep 05 19:51:32.391755 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.391701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/617ea30d-e653-4a6f-882b-3ea090eee8ca-tls-certs\") pod \"kube-auth-proxy-84fbbb6876-9dwpv\" (UID: \"617ea30d-e653-4a6f-882b-3ea090eee8ca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" Sep 05 19:51:32.391951 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:51:32.391871 2578 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 05 19:51:32.391951 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:51:32.391949 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617ea30d-e653-4a6f-882b-3ea090eee8ca-tls-certs podName:617ea30d-e653-4a6f-882b-3ea090eee8ca nodeName:}" failed. No retries permitted until 2026-09-05 19:51:32.891931704 +0000 UTC m=+513.377232573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/617ea30d-e653-4a6f-882b-3ea090eee8ca-tls-certs") pod "kube-auth-proxy-84fbbb6876-9dwpv" (UID: "617ea30d-e653-4a6f-882b-3ea090eee8ca") : secret "kube-auth-proxy-tls" not found Sep 05 19:51:32.392026 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.391873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/617ea30d-e653-4a6f-882b-3ea090eee8ca-tmp\") pod \"kube-auth-proxy-84fbbb6876-9dwpv\" (UID: \"617ea30d-e653-4a6f-882b-3ea090eee8ca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" Sep 05 19:51:32.392026 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.392016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f9tg\" (UniqueName: \"kubernetes.io/projected/617ea30d-e653-4a6f-882b-3ea090eee8ca-kube-api-access-9f9tg\") pod \"kube-auth-proxy-84fbbb6876-9dwpv\" (UID: \"617ea30d-e653-4a6f-882b-3ea090eee8ca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" Sep 05 19:51:32.394100 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.394073 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/617ea30d-e653-4a6f-882b-3ea090eee8ca-tmp\") pod \"kube-auth-proxy-84fbbb6876-9dwpv\" (UID: \"617ea30d-e653-4a6f-882b-3ea090eee8ca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" Sep 05 19:51:32.405074 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.405047 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f9tg\" (UniqueName: \"kubernetes.io/projected/617ea30d-e653-4a6f-882b-3ea090eee8ca-kube-api-access-9f9tg\") pod \"kube-auth-proxy-84fbbb6876-9dwpv\" (UID: \"617ea30d-e653-4a6f-882b-3ea090eee8ca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" Sep 05 19:51:32.897177 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.897141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/617ea30d-e653-4a6f-882b-3ea090eee8ca-tls-certs\") pod \"kube-auth-proxy-84fbbb6876-9dwpv\" (UID: \"617ea30d-e653-4a6f-882b-3ea090eee8ca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" Sep 05 19:51:32.899575 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:32.899544 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/617ea30d-e653-4a6f-882b-3ea090eee8ca-tls-certs\") pod \"kube-auth-proxy-84fbbb6876-9dwpv\" (UID: \"617ea30d-e653-4a6f-882b-3ea090eee8ca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" Sep 05 19:51:33.098639 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:33.098603 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" Sep 05 19:51:33.389718 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:33.389628 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv"] Sep 05 19:51:33.392389 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:51:33.392365 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617ea30d_e653_4a6f_882b_3ea090eee8ca.slice/crio-1651b2756d64e181422714510fb09510cdd41ba0bd1ca5f07ec5e30430733519 WatchSource:0}: Error finding container 1651b2756d64e181422714510fb09510cdd41ba0bd1ca5f07ec5e30430733519: Status 404 returned error can't find the container with id 1651b2756d64e181422714510fb09510cdd41ba0bd1ca5f07ec5e30430733519 Sep 05 19:51:34.032041 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:34.032000 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" event={"ID":"617ea30d-e653-4a6f-882b-3ea090eee8ca","Type":"ContainerStarted","Data":"1651b2756d64e181422714510fb09510cdd41ba0bd1ca5f07ec5e30430733519"} Sep 05 19:51:37.047728 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:37.047682 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" event={"ID":"617ea30d-e653-4a6f-882b-3ea090eee8ca","Type":"ContainerStarted","Data":"f60227c3109cc9e7fe23ecc0bf7e7baf26e54e3696a96e649092a5dfc240aaa3"} Sep 05 19:51:37.065249 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:37.065203 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84fbbb6876-9dwpv" podStartSLOduration=2.449782522 podStartE2EDuration="5.06518736s" totalImagesPullingTime="2.615404838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:37.064823471 +0000 UTC m=+517.550124348" watchObservedRunningTime="2026-09-05 19:51:37.06518736 +0000 UTC m=+517.550488233" Sep 05 19:51:43.027202 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:51:43.027170 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-tscz7" Sep 05 19:52:59.974886 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:52:59.974850 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 19:52:59.975702 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:52:59.975683 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 19:54:11.951764 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:11.951724 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2djr8"] Sep 05 19:54:11.954986 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:11.954969 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" Sep 05 19:54:11.957992 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:11.957970 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 05 19:54:11.959129 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:11.959109 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m9t9m\"" Sep 05 19:54:11.959242 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:11.959135 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 05 19:54:11.959242 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:11.959144 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 05 19:54:11.965361 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:11.965333 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2djr8"] Sep 05 19:54:12.043129 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.043094 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2djr8"] Sep 05 19:54:12.057933 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.057900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79rnq\" (UniqueName: \"kubernetes.io/projected/b3879985-b3d3-43a0-90dc-8edeea16e761-kube-api-access-79rnq\") pod \"limitador-limitador-859c4d96d6-2djr8\" (UID: \"b3879985-b3d3-43a0-90dc-8edeea16e761\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" Sep 05 19:54:12.058098 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.057950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3879985-b3d3-43a0-90dc-8edeea16e761-config-file\") pod \"limitador-limitador-859c4d96d6-2djr8\" (UID: \"b3879985-b3d3-43a0-90dc-8edeea16e761\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" Sep 05 19:54:12.158583 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.158536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79rnq\" (UniqueName: \"kubernetes.io/projected/b3879985-b3d3-43a0-90dc-8edeea16e761-kube-api-access-79rnq\") pod \"limitador-limitador-859c4d96d6-2djr8\" (UID: \"b3879985-b3d3-43a0-90dc-8edeea16e761\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" Sep 05 19:54:12.158761 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.158602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3879985-b3d3-43a0-90dc-8edeea16e761-config-file\") pod \"limitador-limitador-859c4d96d6-2djr8\" (UID: \"b3879985-b3d3-43a0-90dc-8edeea16e761\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" Sep 05 19:54:12.159257 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.159238 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3879985-b3d3-43a0-90dc-8edeea16e761-config-file\") pod \"limitador-limitador-859c4d96d6-2djr8\" (UID: \"b3879985-b3d3-43a0-90dc-8edeea16e761\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" Sep 05 19:54:12.167844 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.167823 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79rnq\" (UniqueName: \"kubernetes.io/projected/b3879985-b3d3-43a0-90dc-8edeea16e761-kube-api-access-79rnq\") pod \"limitador-limitador-859c4d96d6-2djr8\" (UID: \"b3879985-b3d3-43a0-90dc-8edeea16e761\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" Sep 05 19:54:12.265157 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.265121 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" Sep 05 19:54:12.541102 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.541020 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2djr8"] Sep 05 19:54:12.544095 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:54:12.544071 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3879985_b3d3_43a0_90dc_8edeea16e761.slice/crio-e79d8459aeaacf289ad5c2ddebfe7fb7678bcfd7274b1598a046c32e811db07c WatchSource:0}: Error finding container e79d8459aeaacf289ad5c2ddebfe7fb7678bcfd7274b1598a046c32e811db07c: Status 404 returned error can't find the container with id e79d8459aeaacf289ad5c2ddebfe7fb7678bcfd7274b1598a046c32e811db07c Sep 05 19:54:12.711783 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.711748 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" event={"ID":"b3879985-b3d3-43a0-90dc-8edeea16e761","Type":"ContainerStarted","Data":"e79d8459aeaacf289ad5c2ddebfe7fb7678bcfd7274b1598a046c32e811db07c"} Sep 05 19:54:12.729724 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.729689 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-vktpk"] Sep 05 19:54:12.734234 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.734212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-vktpk" Sep 05 19:54:12.736894 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.736872 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2zhs7\"" Sep 05 19:54:12.740157 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.740136 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-vktpk"] Sep 05 19:54:12.762772 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.762741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mlwd\" (UniqueName: \"kubernetes.io/projected/c7fac4bb-6c5c-4c67-9d23-00e26c59a639-kube-api-access-9mlwd\") pod \"authorino-b54df94b4-vktpk\" (UID: \"c7fac4bb-6c5c-4c67-9d23-00e26c59a639\") " pod="kuadrant-system/authorino-b54df94b4-vktpk" Sep 05 19:54:12.864087 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.863998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mlwd\" (UniqueName: \"kubernetes.io/projected/c7fac4bb-6c5c-4c67-9d23-00e26c59a639-kube-api-access-9mlwd\") pod \"authorino-b54df94b4-vktpk\" (UID: \"c7fac4bb-6c5c-4c67-9d23-00e26c59a639\") " pod="kuadrant-system/authorino-b54df94b4-vktpk" Sep 05 19:54:12.876344 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:12.876311 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mlwd\" (UniqueName: \"kubernetes.io/projected/c7fac4bb-6c5c-4c67-9d23-00e26c59a639-kube-api-access-9mlwd\") pod \"authorino-b54df94b4-vktpk\" (UID: \"c7fac4bb-6c5c-4c67-9d23-00e26c59a639\") " pod="kuadrant-system/authorino-b54df94b4-vktpk" Sep 05 19:54:13.044625 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:13.044592 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-vktpk" Sep 05 19:54:13.338457 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:13.338423 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-vktpk"] Sep 05 19:54:13.341595 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:54:13.341568 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7fac4bb_6c5c_4c67_9d23_00e26c59a639.slice/crio-6439b0a70b33dd8906a3a8735db2171e6c7ac6f71c2bcab5c0c0a0c3fe5f8b8a WatchSource:0}: Error finding container 6439b0a70b33dd8906a3a8735db2171e6c7ac6f71c2bcab5c0c0a0c3fe5f8b8a: Status 404 returned error can't find the container with id 6439b0a70b33dd8906a3a8735db2171e6c7ac6f71c2bcab5c0c0a0c3fe5f8b8a Sep 05 19:54:13.719892 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:13.719774 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-vktpk" event={"ID":"c7fac4bb-6c5c-4c67-9d23-00e26c59a639","Type":"ContainerStarted","Data":"6439b0a70b33dd8906a3a8735db2171e6c7ac6f71c2bcab5c0c0a0c3fe5f8b8a"} Sep 05 19:54:15.731549 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:15.731514 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" event={"ID":"b3879985-b3d3-43a0-90dc-8edeea16e761","Type":"ContainerStarted","Data":"e78a8c227c002ee664e3f2a314421f6658acac2c1d1d8f3e5acb7bb1e77bd532"} Sep 05 19:54:15.731989 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:15.731604 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" Sep 05 19:54:15.750314 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:15.750261 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" podStartSLOduration=2.380926321 podStartE2EDuration="4.750244587s" totalImagesPullingTime="2.369318266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:54:15.74892502 +0000 UTC m=+676.234225896" watchObservedRunningTime="2026-09-05 19:54:15.750244587 +0000 UTC m=+676.235545459" Sep 05 19:54:16.368258 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:16.368169 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-vktpk"] Sep 05 19:54:16.738047 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:16.737952 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-vktpk" event={"ID":"c7fac4bb-6c5c-4c67-9d23-00e26c59a639","Type":"ContainerStarted","Data":"4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a"} Sep 05 19:54:16.755699 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:16.755649 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-vktpk" podStartSLOduration=2.048308781 podStartE2EDuration="4.755632994s" totalImagesPullingTime="2.707324213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:54:16.7548434 +0000 UTC m=+677.240144273" watchObservedRunningTime="2026-09-05 19:54:16.755632994 +0000 UTC m=+677.240933867" Sep 05 19:54:17.743194 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:17.743145 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-vktpk" podUID="c7fac4bb-6c5c-4c67-9d23-00e26c59a639" containerName="authorino" containerID="cri-o://4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a" gracePeriod=30 Sep 05 19:54:17.993703 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:17.993638 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-vktpk" Sep 05 19:54:18.118042 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:18.118007 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mlwd\" (UniqueName: \"kubernetes.io/projected/c7fac4bb-6c5c-4c67-9d23-00e26c59a639-kube-api-access-9mlwd\") pod \"c7fac4bb-6c5c-4c67-9d23-00e26c59a639\" (UID: \"c7fac4bb-6c5c-4c67-9d23-00e26c59a639\") " Sep 05 19:54:18.120206 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:18.120181 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fac4bb-6c5c-4c67-9d23-00e26c59a639-kube-api-access-9mlwd" (OuterVolumeSpecName: "kube-api-access-9mlwd") pod "c7fac4bb-6c5c-4c67-9d23-00e26c59a639" (UID: "c7fac4bb-6c5c-4c67-9d23-00e26c59a639"). InnerVolumeSpecName "kube-api-access-9mlwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:54:18.219077 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:18.219034 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mlwd\" (UniqueName: \"kubernetes.io/projected/c7fac4bb-6c5c-4c67-9d23-00e26c59a639-kube-api-access-9mlwd\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:54:18.748931 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:18.748889 2578 generic.go:356] "Generic (PLEG): container finished" podID="c7fac4bb-6c5c-4c67-9d23-00e26c59a639" containerID="4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a" exitCode=0 Sep 05 19:54:18.749346 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:18.748952 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-vktpk" Sep 05 19:54:18.749346 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:18.748973 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-vktpk" event={"ID":"c7fac4bb-6c5c-4c67-9d23-00e26c59a639","Type":"ContainerDied","Data":"4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a"} Sep 05 19:54:18.749346 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:18.749017 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-vktpk" event={"ID":"c7fac4bb-6c5c-4c67-9d23-00e26c59a639","Type":"ContainerDied","Data":"6439b0a70b33dd8906a3a8735db2171e6c7ac6f71c2bcab5c0c0a0c3fe5f8b8a"} Sep 05 19:54:18.749346 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:18.749037 2578 scope.go:117] "RemoveContainer" containerID="4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a" Sep 05 19:54:18.757808 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:18.757781 2578 scope.go:117] "RemoveContainer" containerID="4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a" Sep 05 19:54:18.758093 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:54:18.758075 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a\": container with ID starting with 4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a not found: ID does not exist" containerID="4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a" Sep 05 19:54:18.758165 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:18.758101 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a"} err="failed to get container status \"4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a\": rpc error: code = NotFound desc = could not find container \"4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a\": container with ID starting with 4a7b5663d930e5c0802fa97690b23ae625e919ec5145afe8055de5b95127d14a not found: ID does not exist" Sep 05 19:54:18.770558 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:18.770535 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-vktpk"] Sep 05 19:54:18.774202 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:18.774183 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-vktpk"] Sep 05 19:54:20.011362 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:20.011327 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7fac4bb-6c5c-4c67-9d23-00e26c59a639" path="/var/lib/kubelet/pods/c7fac4bb-6c5c-4c67-9d23-00e26c59a639/volumes" Sep 05 19:54:26.739396 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:26.739363 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" Sep 05 19:54:26.863786 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:26.863743 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2djr8"] Sep 05 19:54:26.863998 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:26.863973 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" podUID="b3879985-b3d3-43a0-90dc-8edeea16e761" containerName="limitador" containerID="cri-o://e78a8c227c002ee664e3f2a314421f6658acac2c1d1d8f3e5acb7bb1e77bd532" gracePeriod=30 Sep 05 19:54:27.795557 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:27.795526 2578 generic.go:356] "Generic (PLEG): container finished" podID="b3879985-b3d3-43a0-90dc-8edeea16e761" containerID="e78a8c227c002ee664e3f2a314421f6658acac2c1d1d8f3e5acb7bb1e77bd532" exitCode=0 Sep 05 19:54:27.795898 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:27.795606 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" event={"ID":"b3879985-b3d3-43a0-90dc-8edeea16e761","Type":"ContainerDied","Data":"e78a8c227c002ee664e3f2a314421f6658acac2c1d1d8f3e5acb7bb1e77bd532"} Sep 05 19:54:27.795898 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:27.795654 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" event={"ID":"b3879985-b3d3-43a0-90dc-8edeea16e761","Type":"ContainerDied","Data":"e79d8459aeaacf289ad5c2ddebfe7fb7678bcfd7274b1598a046c32e811db07c"} Sep 05 19:54:27.795898 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:27.795676 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e79d8459aeaacf289ad5c2ddebfe7fb7678bcfd7274b1598a046c32e811db07c" Sep 05 19:54:27.796756 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:27.796739 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" Sep 05 19:54:27.907742 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:27.907703 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3879985-b3d3-43a0-90dc-8edeea16e761-config-file\") pod \"b3879985-b3d3-43a0-90dc-8edeea16e761\" (UID: \"b3879985-b3d3-43a0-90dc-8edeea16e761\") " Sep 05 19:54:27.907956 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:27.907818 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79rnq\" (UniqueName: \"kubernetes.io/projected/b3879985-b3d3-43a0-90dc-8edeea16e761-kube-api-access-79rnq\") pod \"b3879985-b3d3-43a0-90dc-8edeea16e761\" (UID: \"b3879985-b3d3-43a0-90dc-8edeea16e761\") " Sep 05 19:54:27.908124 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:27.908101 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3879985-b3d3-43a0-90dc-8edeea16e761-config-file" (OuterVolumeSpecName: "config-file") pod "b3879985-b3d3-43a0-90dc-8edeea16e761" (UID: "b3879985-b3d3-43a0-90dc-8edeea16e761"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:54:27.909827 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:27.909806 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3879985-b3d3-43a0-90dc-8edeea16e761-kube-api-access-79rnq" (OuterVolumeSpecName: "kube-api-access-79rnq") pod "b3879985-b3d3-43a0-90dc-8edeea16e761" (UID: "b3879985-b3d3-43a0-90dc-8edeea16e761"). InnerVolumeSpecName "kube-api-access-79rnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:54:28.008498 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:28.008466 2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3879985-b3d3-43a0-90dc-8edeea16e761-config-file\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:54:28.008498 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:28.008495 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79rnq\" (UniqueName: \"kubernetes.io/projected/b3879985-b3d3-43a0-90dc-8edeea16e761-kube-api-access-79rnq\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:54:28.799545 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:28.799516 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2djr8" Sep 05 19:54:28.817857 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:28.817823 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2djr8"] Sep 05 19:54:28.822159 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:28.822131 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2djr8"] Sep 05 19:54:30.011867 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:30.011834 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3879985-b3d3-43a0-90dc-8edeea16e761" path="/var/lib/kubelet/pods/b3879985-b3d3-43a0-90dc-8edeea16e761/volumes" Sep 05 19:54:32.082100 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.082067 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x"] Sep 05 19:54:32.082451 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.082376 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3879985-b3d3-43a0-90dc-8edeea16e761" containerName="limitador" Sep 05 19:54:32.082451 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.082387 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3879985-b3d3-43a0-90dc-8edeea16e761" containerName="limitador" Sep 05 19:54:32.082451 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.082398 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7fac4bb-6c5c-4c67-9d23-00e26c59a639" containerName="authorino" Sep 05 19:54:32.082451 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.082403 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7fac4bb-6c5c-4c67-9d23-00e26c59a639" containerName="authorino" Sep 05 19:54:32.082451 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.082452 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7fac4bb-6c5c-4c67-9d23-00e26c59a639" containerName="authorino" Sep 05 19:54:32.082607 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.082461 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3879985-b3d3-43a0-90dc-8edeea16e761" containerName="limitador" Sep 05 19:54:32.085524 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.085506 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" Sep 05 19:54:32.088593 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.088572 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 19:54:32.088941 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.088924 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 05 19:54:32.089031 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.088991 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w44pd\"" Sep 05 19:54:32.089764 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.089749 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 05 19:54:32.097252 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.097231 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x"] Sep 05 19:54:32.243274 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.243231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bbaffa32-4a72-4204-bb51-e4bd03a27113-data\") pod \"postgres-69fb4594cf-x9s9x\" (UID: \"bbaffa32-4a72-4204-bb51-e4bd03a27113\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" Sep 05 19:54:32.243448 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.243352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqgps\" (UniqueName: \"kubernetes.io/projected/bbaffa32-4a72-4204-bb51-e4bd03a27113-kube-api-access-vqgps\") pod \"postgres-69fb4594cf-x9s9x\" (UID: \"bbaffa32-4a72-4204-bb51-e4bd03a27113\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" Sep 05 19:54:32.344362 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.344269 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqgps\" (UniqueName: \"kubernetes.io/projected/bbaffa32-4a72-4204-bb51-e4bd03a27113-kube-api-access-vqgps\") pod \"postgres-69fb4594cf-x9s9x\" (UID: \"bbaffa32-4a72-4204-bb51-e4bd03a27113\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" Sep 05 19:54:32.344362 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.344314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bbaffa32-4a72-4204-bb51-e4bd03a27113-data\") pod \"postgres-69fb4594cf-x9s9x\" (UID: \"bbaffa32-4a72-4204-bb51-e4bd03a27113\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" Sep 05 19:54:32.344691 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.344672 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bbaffa32-4a72-4204-bb51-e4bd03a27113-data\") pod \"postgres-69fb4594cf-x9s9x\" (UID: \"bbaffa32-4a72-4204-bb51-e4bd03a27113\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" Sep 05 19:54:32.354195 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.354166 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqgps\" (UniqueName: \"kubernetes.io/projected/bbaffa32-4a72-4204-bb51-e4bd03a27113-kube-api-access-vqgps\") pod \"postgres-69fb4594cf-x9s9x\" (UID: \"bbaffa32-4a72-4204-bb51-e4bd03a27113\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" Sep 05 19:54:32.395225 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.395188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" Sep 05 19:54:32.667688 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.667650 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x"] Sep 05 19:54:32.670438 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:54:32.670410 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbaffa32_4a72_4204_bb51_e4bd03a27113.slice/crio-e7a49f8664a8a11d379b6e702758673e524afffd3b1dc38cba1b52dc01753a36 WatchSource:0}: Error finding container e7a49f8664a8a11d379b6e702758673e524afffd3b1dc38cba1b52dc01753a36: Status 404 returned error can't find the container with id e7a49f8664a8a11d379b6e702758673e524afffd3b1dc38cba1b52dc01753a36 Sep 05 19:54:32.818242 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:32.818207 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" event={"ID":"bbaffa32-4a72-4204-bb51-e4bd03a27113","Type":"ContainerStarted","Data":"e7a49f8664a8a11d379b6e702758673e524afffd3b1dc38cba1b52dc01753a36"} Sep 05 19:54:37.855599 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:37.855551 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" event={"ID":"bbaffa32-4a72-4204-bb51-e4bd03a27113","Type":"ContainerStarted","Data":"96ac228cf32b3cdc945a8d41b33466d809de995836efb46225b4b15bbcb4b0b2"} Sep 05 19:54:37.856017 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:37.855729 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" Sep 05 19:54:37.874325 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:37.874271 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" podStartSLOduration=1.096324677 podStartE2EDuration="5.874254475s" totalImagesPullingTime="4.777929798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:54:37.873019311 +0000 UTC m=+698.358320184" watchObservedRunningTime="2026-09-05 19:54:37.874254475 +0000 UTC m=+698.359555348" Sep 05 19:54:43.889978 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:43.889940 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x9s9x" Sep 05 19:54:44.423254 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.423218 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-8dz42"] Sep 05 19:54:44.429403 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.429384 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-8dz42" Sep 05 19:54:44.432454 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.432430 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 05 19:54:44.433300 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.433275 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2zhs7\"" Sep 05 19:54:44.433410 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.433285 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 05 19:54:44.433951 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.433926 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-8dz42"] Sep 05 19:54:44.553865 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.553826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqpf8\" (UniqueName: \"kubernetes.io/projected/24c687e4-a7d0-472b-8ded-40ad12c63643-kube-api-access-tqpf8\") pod \"authorino-5dc7ccb5f-8dz42\" (UID: \"24c687e4-a7d0-472b-8ded-40ad12c63643\") " pod="kuadrant-system/authorino-5dc7ccb5f-8dz42" Sep 05 19:54:44.638844 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.638809 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-8dz42"] Sep 05 19:54:44.639085 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:54:44.639063 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tqpf8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-8dz42" podUID="24c687e4-a7d0-472b-8ded-40ad12c63643" Sep 05 19:54:44.654634 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.654602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqpf8\" (UniqueName: \"kubernetes.io/projected/24c687e4-a7d0-472b-8ded-40ad12c63643-kube-api-access-tqpf8\") pod \"authorino-5dc7ccb5f-8dz42\" (UID: \"24c687e4-a7d0-472b-8ded-40ad12c63643\") " pod="kuadrant-system/authorino-5dc7ccb5f-8dz42" Sep 05 19:54:44.665536 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.665511 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7db9bf948c-62gdw"] Sep 05 19:54:44.668154 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.668129 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqpf8\" (UniqueName: \"kubernetes.io/projected/24c687e4-a7d0-472b-8ded-40ad12c63643-kube-api-access-tqpf8\") pod \"authorino-5dc7ccb5f-8dz42\" (UID: \"24c687e4-a7d0-472b-8ded-40ad12c63643\") " pod="kuadrant-system/authorino-5dc7ccb5f-8dz42" Sep 05 19:54:44.669560 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.669543 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7db9bf948c-62gdw" Sep 05 19:54:44.676527 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.676471 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7db9bf948c-62gdw"] Sep 05 19:54:44.736034 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.736000 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7db9bf948c-62gdw"] Sep 05 19:54:44.736272 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:54:44.736254 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6tqtq], unattached volumes=[], failed to process volumes=[kube-api-access-6tqtq]: context canceled" pod="kuadrant-system/authorino-7db9bf948c-62gdw" podUID="99649f6d-4371-431f-a950-71c45f1760aa" Sep 05 19:54:44.857164 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.857125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tqtq\" (UniqueName: \"kubernetes.io/projected/99649f6d-4371-431f-a950-71c45f1760aa-kube-api-access-6tqtq\") pod \"authorino-7db9bf948c-62gdw\" (UID: \"99649f6d-4371-431f-a950-71c45f1760aa\") " pod="kuadrant-system/authorino-7db9bf948c-62gdw" Sep 05 19:54:44.886975 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.886948 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7db9bf948c-62gdw" Sep 05 19:54:44.887134 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.886950 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-8dz42" Sep 05 19:54:44.894347 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.894327 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7db9bf948c-62gdw" Sep 05 19:54:44.896995 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.896980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-8dz42" Sep 05 19:54:44.957744 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.957641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tqtq\" (UniqueName: \"kubernetes.io/projected/99649f6d-4371-431f-a950-71c45f1760aa-kube-api-access-6tqtq\") pod \"authorino-7db9bf948c-62gdw\" (UID: \"99649f6d-4371-431f-a950-71c45f1760aa\") " pod="kuadrant-system/authorino-7db9bf948c-62gdw" Sep 05 19:54:44.970512 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:44.970487 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tqtq\" (UniqueName: \"kubernetes.io/projected/99649f6d-4371-431f-a950-71c45f1760aa-kube-api-access-6tqtq\") pod \"authorino-7db9bf948c-62gdw\" (UID: \"99649f6d-4371-431f-a950-71c45f1760aa\") " pod="kuadrant-system/authorino-7db9bf948c-62gdw" Sep 05 19:54:45.058574 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:45.058536 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqpf8\" (UniqueName: \"kubernetes.io/projected/24c687e4-a7d0-472b-8ded-40ad12c63643-kube-api-access-tqpf8\") pod \"24c687e4-a7d0-472b-8ded-40ad12c63643\" (UID: \"24c687e4-a7d0-472b-8ded-40ad12c63643\") " Sep 05 19:54:45.060610 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:45.060575 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c687e4-a7d0-472b-8ded-40ad12c63643-kube-api-access-tqpf8" (OuterVolumeSpecName: "kube-api-access-tqpf8") pod "24c687e4-a7d0-472b-8ded-40ad12c63643" (UID: "24c687e4-a7d0-472b-8ded-40ad12c63643"). InnerVolumeSpecName "kube-api-access-tqpf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:54:45.159359 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:45.159321 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tqtq\" (UniqueName: \"kubernetes.io/projected/99649f6d-4371-431f-a950-71c45f1760aa-kube-api-access-6tqtq\") pod \"99649f6d-4371-431f-a950-71c45f1760aa\" (UID: \"99649f6d-4371-431f-a950-71c45f1760aa\") " Sep 05 19:54:45.159560 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:45.159549 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqpf8\" (UniqueName: \"kubernetes.io/projected/24c687e4-a7d0-472b-8ded-40ad12c63643-kube-api-access-tqpf8\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:54:45.161391 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:45.161352 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99649f6d-4371-431f-a950-71c45f1760aa-kube-api-access-6tqtq" (OuterVolumeSpecName: "kube-api-access-6tqtq") pod "99649f6d-4371-431f-a950-71c45f1760aa" (UID: "99649f6d-4371-431f-a950-71c45f1760aa"). InnerVolumeSpecName "kube-api-access-6tqtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:54:45.261009 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:45.260970 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tqtq\" (UniqueName: \"kubernetes.io/projected/99649f6d-4371-431f-a950-71c45f1760aa-kube-api-access-6tqtq\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:54:45.891198 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:45.891164 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-8dz42" Sep 05 19:54:45.891378 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:45.891167 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7db9bf948c-62gdw" Sep 05 19:54:45.923665 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:45.923608 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-8dz42"] Sep 05 19:54:45.926080 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:45.926054 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-8dz42"] Sep 05 19:54:45.946913 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:45.946876 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7db9bf948c-62gdw"] Sep 05 19:54:45.950792 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:45.950762 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7db9bf948c-62gdw"] Sep 05 19:54:46.011658 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:46.011626 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c687e4-a7d0-472b-8ded-40ad12c63643" path="/var/lib/kubelet/pods/24c687e4-a7d0-472b-8ded-40ad12c63643/volumes" Sep 05 19:54:46.011883 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:46.011870 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99649f6d-4371-431f-a950-71c45f1760aa" path="/var/lib/kubelet/pods/99649f6d-4371-431f-a950-71c45f1760aa/volumes" Sep 05 19:54:51.112548 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.112507 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67b48bfc7b-lzd56"] Sep 05 19:54:51.268311 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.268269 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67b48bfc7b-lzd56"] Sep 05 19:54:51.268473 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.268392 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:51.271284 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.271246 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 05 19:54:51.271284 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.271276 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 05 19:54:51.271562 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.271321 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d4vmp\"" Sep 05 19:54:51.271562 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.271356 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 05 19:54:51.274128 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.274069 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-7l6k5"] Sep 05 19:54:51.294744 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.294712 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-7l6k5"] Sep 05 19:54:51.294947 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.294847 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:51.313152 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.313114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2sq\" (UniqueName: \"kubernetes.io/projected/d1e1b305-ed05-47d2-93eb-8514960be1f5-kube-api-access-vx2sq\") pod \"maas-controller-67b48bfc7b-lzd56\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:51.313307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.313163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-metrics-certs\") pod \"maas-controller-7686cbc6c4-7l6k5\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:51.313307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.313179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-metrics-certs\") pod \"maas-controller-67b48bfc7b-lzd56\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:51.313307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.313212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-webhook-cert\") pod \"maas-controller-7686cbc6c4-7l6k5\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:51.313307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.313251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w68x8\" (UniqueName: \"kubernetes.io/projected/b9221c89-4902-4f7f-ae35-fd159704e009-kube-api-access-w68x8\") pod \"maas-controller-7686cbc6c4-7l6k5\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:51.313307 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.313266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-webhook-cert\") pod \"maas-controller-67b48bfc7b-lzd56\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:51.414044 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.413954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68x8\" (UniqueName: \"kubernetes.io/projected/b9221c89-4902-4f7f-ae35-fd159704e009-kube-api-access-w68x8\") pod \"maas-controller-7686cbc6c4-7l6k5\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:51.414044 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.413992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-webhook-cert\") pod \"maas-controller-67b48bfc7b-lzd56\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:51.414044 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.414037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2sq\" (UniqueName: \"kubernetes.io/projected/d1e1b305-ed05-47d2-93eb-8514960be1f5-kube-api-access-vx2sq\") pod \"maas-controller-67b48bfc7b-lzd56\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:51.414335 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.414068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-metrics-certs\") pod \"maas-controller-7686cbc6c4-7l6k5\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:51.414335 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.414086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-metrics-certs\") pod \"maas-controller-67b48bfc7b-lzd56\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:51.414335 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:54:51.414148 2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 05 19:54:51.414335 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.414204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-webhook-cert\") pod \"maas-controller-7686cbc6c4-7l6k5\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:51.414335 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:54:51.414230 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-webhook-cert podName:d1e1b305-ed05-47d2-93eb-8514960be1f5 nodeName:}" failed. No retries permitted until 2026-09-05 19:54:51.914209569 +0000 UTC m=+712.399510439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-webhook-cert") pod "maas-controller-67b48bfc7b-lzd56" (UID: "d1e1b305-ed05-47d2-93eb-8514960be1f5") : secret "maas-controller-webhook-cert" not found Sep 05 19:54:51.414335 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:54:51.414264 2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 05 19:54:51.414559 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:54:51.414349 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-webhook-cert podName:b9221c89-4902-4f7f-ae35-fd159704e009 nodeName:}" failed. No retries permitted until 2026-09-05 19:54:51.914331983 +0000 UTC m=+712.399632836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-webhook-cert") pod "maas-controller-7686cbc6c4-7l6k5" (UID: "b9221c89-4902-4f7f-ae35-fd159704e009") : secret "maas-controller-webhook-cert" not found Sep 05 19:54:51.416630 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.416598 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-metrics-certs\") pod \"maas-controller-7686cbc6c4-7l6k5\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:51.416809 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.416665 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-metrics-certs\") pod \"maas-controller-67b48bfc7b-lzd56\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:51.424333 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.424305 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2sq\" (UniqueName: \"kubernetes.io/projected/d1e1b305-ed05-47d2-93eb-8514960be1f5-kube-api-access-vx2sq\") pod \"maas-controller-67b48bfc7b-lzd56\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:51.424434 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.424305 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w68x8\" (UniqueName: \"kubernetes.io/projected/b9221c89-4902-4f7f-ae35-fd159704e009-kube-api-access-w68x8\") pod \"maas-controller-7686cbc6c4-7l6k5\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:51.918623 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.918587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-webhook-cert\") pod \"maas-controller-7686cbc6c4-7l6k5\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:51.918845 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.918648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-webhook-cert\") pod \"maas-controller-67b48bfc7b-lzd56\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:51.921099 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.921065 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-webhook-cert\") pod \"maas-controller-7686cbc6c4-7l6k5\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:51.921341 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:51.921318 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-webhook-cert\") pod \"maas-controller-67b48bfc7b-lzd56\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:52.178724 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:52.178597 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:52.206705 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:52.206669 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:52.454860 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:52.454833 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67b48bfc7b-lzd56"] Sep 05 19:54:52.457153 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:54:52.457121 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e1b305_ed05_47d2_93eb_8514960be1f5.slice/crio-856e94f8b89ea2958a644a018e739ffc97646faf3da0cbee3972dc7cdfe80244 WatchSource:0}: Error finding container 856e94f8b89ea2958a644a018e739ffc97646faf3da0cbee3972dc7cdfe80244: Status 404 returned error can't find the container with id 856e94f8b89ea2958a644a018e739ffc97646faf3da0cbee3972dc7cdfe80244 Sep 05 19:54:52.490522 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:52.490493 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-7l6k5"] Sep 05 19:54:52.493121 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:54:52.493097 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9221c89_4902_4f7f_ae35_fd159704e009.slice/crio-aa64845621a09e14270800b73fad79f2121d68a5892208a64a8e367035a01e69 WatchSource:0}: Error finding container aa64845621a09e14270800b73fad79f2121d68a5892208a64a8e367035a01e69: Status 404 returned error can't find the container with id aa64845621a09e14270800b73fad79f2121d68a5892208a64a8e367035a01e69 Sep 05 19:54:52.933511 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:52.933467 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" event={"ID":"b9221c89-4902-4f7f-ae35-fd159704e009","Type":"ContainerStarted","Data":"aa64845621a09e14270800b73fad79f2121d68a5892208a64a8e367035a01e69"} Sep 05 19:54:52.935122 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:52.935082 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b48bfc7b-lzd56" event={"ID":"d1e1b305-ed05-47d2-93eb-8514960be1f5","Type":"ContainerStarted","Data":"856e94f8b89ea2958a644a018e739ffc97646faf3da0cbee3972dc7cdfe80244"} Sep 05 19:54:55.953178 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:55.953135 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" event={"ID":"b9221c89-4902-4f7f-ae35-fd159704e009","Type":"ContainerStarted","Data":"8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34"} Sep 05 19:54:55.953642 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:55.953359 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:54:55.954689 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:55.954664 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b48bfc7b-lzd56" event={"ID":"d1e1b305-ed05-47d2-93eb-8514960be1f5","Type":"ContainerStarted","Data":"7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0"} Sep 05 19:54:55.954823 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:55.954714 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:54:55.972506 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:55.972454 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" podStartSLOduration=2.540498695 podStartE2EDuration="4.972439497s" totalImagesPullingTime="2.431940802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:54:55.972090274 +0000 UTC m=+716.457391147" watchObservedRunningTime="2026-09-05 19:54:55.972439497 +0000 UTC m=+716.457740369" Sep 05 19:54:55.989989 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:54:55.989937 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67b48bfc7b-lzd56" podStartSLOduration=2.5145627619999997 podStartE2EDuration="4.989922081s" totalImagesPullingTime="2.475359319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:54:55.989384513 +0000 UTC m=+716.474685398" watchObservedRunningTime="2026-09-05 19:54:55.989922081 +0000 UTC m=+716.475222953" Sep 05 19:55:06.962373 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:06.962338 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:55:06.962820 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:06.962555 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 19:55:07.024334 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:07.024298 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67b48bfc7b-lzd56"] Sep 05 19:55:07.024520 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:07.024494 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-67b48bfc7b-lzd56" podUID="d1e1b305-ed05-47d2-93eb-8514960be1f5" containerName="manager" containerID="cri-o://7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0" gracePeriod=30 Sep 05 19:55:07.265537 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:07.265509 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:55:07.354824 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:07.354758 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-webhook-cert\") pod \"d1e1b305-ed05-47d2-93eb-8514960be1f5\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " Sep 05 19:55:07.354990 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:07.354944 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx2sq\" (UniqueName: \"kubernetes.io/projected/d1e1b305-ed05-47d2-93eb-8514960be1f5-kube-api-access-vx2sq\") pod \"d1e1b305-ed05-47d2-93eb-8514960be1f5\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " Sep 05 19:55:07.354990 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:07.354977 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-metrics-certs\") pod \"d1e1b305-ed05-47d2-93eb-8514960be1f5\" (UID: \"d1e1b305-ed05-47d2-93eb-8514960be1f5\") " Sep 05 19:55:07.357092 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:07.357057 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d1e1b305-ed05-47d2-93eb-8514960be1f5" (UID: "d1e1b305-ed05-47d2-93eb-8514960be1f5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:55:07.357204 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:07.357113 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d1e1b305-ed05-47d2-93eb-8514960be1f5" (UID: "d1e1b305-ed05-47d2-93eb-8514960be1f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:55:07.357204 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:07.357126 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e1b305-ed05-47d2-93eb-8514960be1f5-kube-api-access-vx2sq" (OuterVolumeSpecName: "kube-api-access-vx2sq") pod "d1e1b305-ed05-47d2-93eb-8514960be1f5" (UID: "d1e1b305-ed05-47d2-93eb-8514960be1f5"). InnerVolumeSpecName "kube-api-access-vx2sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:55:07.456071 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:07.456032 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx2sq\" (UniqueName: \"kubernetes.io/projected/d1e1b305-ed05-47d2-93eb-8514960be1f5-kube-api-access-vx2sq\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:55:07.456071 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:07.456063 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:55:07.456071 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:07.456072 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1e1b305-ed05-47d2-93eb-8514960be1f5-webhook-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:55:08.014133 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:08.014099 2578 generic.go:356] "Generic (PLEG): container finished" podID="d1e1b305-ed05-47d2-93eb-8514960be1f5" containerID="7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0" exitCode=0 Sep 05 19:55:08.014527 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:08.014140 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b48bfc7b-lzd56" Sep 05 19:55:08.014527 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:08.014175 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b48bfc7b-lzd56" event={"ID":"d1e1b305-ed05-47d2-93eb-8514960be1f5","Type":"ContainerDied","Data":"7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0"} Sep 05 19:55:08.014527 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:08.014235 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b48bfc7b-lzd56" event={"ID":"d1e1b305-ed05-47d2-93eb-8514960be1f5","Type":"ContainerDied","Data":"856e94f8b89ea2958a644a018e739ffc97646faf3da0cbee3972dc7cdfe80244"} Sep 05 19:55:08.014527 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:08.014257 2578 scope.go:117] "RemoveContainer" containerID="7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0" Sep 05 19:55:08.024007 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:08.023984 2578 scope.go:117] "RemoveContainer" containerID="7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0" Sep 05 19:55:08.024520 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:55:08.024499 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0\": container with ID starting with 7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0 not found: ID does not exist" containerID="7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0" Sep 05 19:55:08.024616 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:08.024528 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0"} err="failed to get container status \"7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0\": rpc error: code = NotFound desc = could not find container \"7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0\": container with ID starting with 7096852725e46cd0c3b098de345e591933c89521bb4449056d75df04222dc4b0 not found: ID does not exist" Sep 05 19:55:08.039681 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:08.039652 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67b48bfc7b-lzd56"] Sep 05 19:55:08.045890 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:08.045862 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67b48bfc7b-lzd56"] Sep 05 19:55:10.013592 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:10.013550 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1e1b305-ed05-47d2-93eb-8514960be1f5" path="/var/lib/kubelet/pods/d1e1b305-ed05-47d2-93eb-8514960be1f5/volumes" Sep 05 19:55:29.264452 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.264423 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr"] Sep 05 19:55:29.264928 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.264721 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e1b305-ed05-47d2-93eb-8514960be1f5" containerName="manager" Sep 05 19:55:29.264928 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.264732 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e1b305-ed05-47d2-93eb-8514960be1f5" containerName="manager" Sep 05 19:55:29.264928 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.264811 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e1b305-ed05-47d2-93eb-8514960be1f5" containerName="manager" Sep 05 19:55:29.270582 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.270560 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.275368 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.275343 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 05 19:55:29.275499 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.275345 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 05 19:55:29.275499 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.275354 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p9ckr\"" Sep 05 19:55:29.284264 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.284237 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr"] Sep 05 19:55:29.447929 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.447894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-metrics-certs\") pod \"maas-api-7b97c6dbfb-n4qfr\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.447929 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.447933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-maas-api-tls\") pod \"maas-api-7b97c6dbfb-n4qfr\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.448147 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.447953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkhtr\" (UniqueName: \"kubernetes.io/projected/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-kube-api-access-bkhtr\") pod \"maas-api-7b97c6dbfb-n4qfr\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.448147 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.448020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-openshift-service-ca\") pod \"maas-api-7b97c6dbfb-n4qfr\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.548826 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.548716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-maas-api-tls\") pod \"maas-api-7b97c6dbfb-n4qfr\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.548826 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.548754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkhtr\" (UniqueName: \"kubernetes.io/projected/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-kube-api-access-bkhtr\") pod \"maas-api-7b97c6dbfb-n4qfr\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.548826 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.548793 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-openshift-service-ca\") pod \"maas-api-7b97c6dbfb-n4qfr\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.549116 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.548929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-metrics-certs\") pod \"maas-api-7b97c6dbfb-n4qfr\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.549514 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.549482 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-openshift-service-ca\") pod \"maas-api-7b97c6dbfb-n4qfr\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.551238 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.551214 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-metrics-certs\") pod \"maas-api-7b97c6dbfb-n4qfr\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.551387 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.551367 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-maas-api-tls\") pod \"maas-api-7b97c6dbfb-n4qfr\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.562002 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.561976 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkhtr\" (UniqueName: \"kubernetes.io/projected/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-kube-api-access-bkhtr\") pod \"maas-api-7b97c6dbfb-n4qfr\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.580682 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.580642 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:29.859292 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.859179 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr"] Sep 05 19:55:29.861947 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:55:29.861921 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a6cb41_08fa_45f0_9ebf_89b5c8c9b996.slice/crio-917c904a03ab2b5b306a663d19f9d1d26c97d12d501362f9d206176d97baba0e WatchSource:0}: Error finding container 917c904a03ab2b5b306a663d19f9d1d26c97d12d501362f9d206176d97baba0e: Status 404 returned error can't find the container with id 917c904a03ab2b5b306a663d19f9d1d26c97d12d501362f9d206176d97baba0e Sep 05 19:55:29.894216 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.894182 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-s75vh"] Sep 05 19:55:29.898724 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.898706 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" Sep 05 19:55:29.901921 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.901895 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-znz9n\"" Sep 05 19:55:29.902165 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.902149 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 05 19:55:29.907543 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:29.907519 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-s75vh"] Sep 05 19:55:30.053625 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:30.053587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6dhd\" (UniqueName: \"kubernetes.io/projected/efeff32f-d46f-42d4-946f-a141c51b5ed4-kube-api-access-l6dhd\") pod \"payload-processing-5b54b7fbb7-s75vh\" (UID: \"efeff32f-d46f-42d4-946f-a141c51b5ed4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" Sep 05 19:55:30.053625 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:30.053625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeff32f-d46f-42d4-946f-a141c51b5ed4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-s75vh\" (UID: \"efeff32f-d46f-42d4-946f-a141c51b5ed4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" Sep 05 19:55:30.119551 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:30.119459 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" event={"ID":"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996","Type":"ContainerStarted","Data":"917c904a03ab2b5b306a663d19f9d1d26c97d12d501362f9d206176d97baba0e"} Sep 05 19:55:30.154180 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:30.154143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6dhd\" (UniqueName: \"kubernetes.io/projected/efeff32f-d46f-42d4-946f-a141c51b5ed4-kube-api-access-l6dhd\") pod \"payload-processing-5b54b7fbb7-s75vh\" (UID: \"efeff32f-d46f-42d4-946f-a141c51b5ed4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" Sep 05 19:55:30.154180 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:30.154191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeff32f-d46f-42d4-946f-a141c51b5ed4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-s75vh\" (UID: \"efeff32f-d46f-42d4-946f-a141c51b5ed4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" Sep 05 19:55:30.155095 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:30.155073 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeff32f-d46f-42d4-946f-a141c51b5ed4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-s75vh\" (UID: \"efeff32f-d46f-42d4-946f-a141c51b5ed4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" Sep 05 19:55:30.166217 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:30.166184 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6dhd\" (UniqueName: \"kubernetes.io/projected/efeff32f-d46f-42d4-946f-a141c51b5ed4-kube-api-access-l6dhd\") pod \"payload-processing-5b54b7fbb7-s75vh\" (UID: \"efeff32f-d46f-42d4-946f-a141c51b5ed4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" Sep 05 19:55:30.209672 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:30.209629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" Sep 05 19:55:30.481260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:30.481228 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-s75vh"] Sep 05 19:55:30.483564 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:55:30.483540 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefeff32f_d46f_42d4_946f_a141c51b5ed4.slice/crio-51828d89491790cebcd2de675d836d707c172bf1c5c72b6e428d261442caa689 WatchSource:0}: Error finding container 51828d89491790cebcd2de675d836d707c172bf1c5c72b6e428d261442caa689: Status 404 returned error can't find the container with id 51828d89491790cebcd2de675d836d707c172bf1c5c72b6e428d261442caa689 Sep 05 19:55:31.129791 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:31.129746 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" event={"ID":"efeff32f-d46f-42d4-946f-a141c51b5ed4","Type":"ContainerStarted","Data":"51828d89491790cebcd2de675d836d707c172bf1c5c72b6e428d261442caa689"} Sep 05 19:55:32.136436 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:32.136388 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" event={"ID":"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996","Type":"ContainerStarted","Data":"92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa"} Sep 05 19:55:32.136841 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:32.136466 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:32.159468 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:32.159419 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" podStartSLOduration=1.429126462 podStartE2EDuration="3.159405256s" totalImagesPullingTime="1.730278794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:55:32.156951067 +0000 UTC m=+752.642252046" watchObservedRunningTime="2026-09-05 19:55:32.159405256 +0000 UTC m=+752.644706129" Sep 05 19:55:33.142716 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:33.142682 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" event={"ID":"efeff32f-d46f-42d4-946f-a141c51b5ed4","Type":"ContainerStarted","Data":"12c671f9ca1f4d3fa90f01b3db69a7887059a10f9b3951b667ae3504109882c6"} Sep 05 19:55:33.143115 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:33.142813 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" Sep 05 19:55:33.163893 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:33.163835 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" podStartSLOduration=2.56327216 podStartE2EDuration="4.163819726s" totalImagesPullingTime="1.600547566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:55:33.162296814 +0000 UTC m=+753.647597689" watchObservedRunningTime="2026-09-05 19:55:33.163819726 +0000 UTC m=+753.649120590" Sep 05 19:55:38.148258 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:38.148227 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 19:55:44.150135 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:55:44.150101 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-s75vh" Sep 05 19:56:14.027263 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.027220 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2"] Sep 05 19:56:14.034874 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.034846 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.038018 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.037994 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dnwjp\"" Sep 05 19:56:14.038324 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.037998 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 05 19:56:14.038461 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.038043 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 05 19:56:14.039150 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.039129 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 05 19:56:14.040896 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.040872 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2"] Sep 05 19:56:14.139819 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.139761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09074706-f21e-40dc-b564-fe4930e934ad-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.139819 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.139825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09074706-f21e-40dc-b564-fe4930e934ad-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.140030 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.139847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fsqz\" (UniqueName: \"kubernetes.io/projected/09074706-f21e-40dc-b564-fe4930e934ad-kube-api-access-2fsqz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.140030 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.139867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09074706-f21e-40dc-b564-fe4930e934ad-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.140030 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.139881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09074706-f21e-40dc-b564-fe4930e934ad-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.140030 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.140002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09074706-f21e-40dc-b564-fe4930e934ad-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.240647 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.240606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09074706-f21e-40dc-b564-fe4930e934ad-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.240860 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.240668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09074706-f21e-40dc-b564-fe4930e934ad-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.240860 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.240780 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09074706-f21e-40dc-b564-fe4930e934ad-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.240860 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.240835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fsqz\" (UniqueName: \"kubernetes.io/projected/09074706-f21e-40dc-b564-fe4930e934ad-kube-api-access-2fsqz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.240999 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.240862 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09074706-f21e-40dc-b564-fe4930e934ad-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.240999 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.240881 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09074706-f21e-40dc-b564-fe4930e934ad-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.241102 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.241025 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09074706-f21e-40dc-b564-fe4930e934ad-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.241157 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.241142 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09074706-f21e-40dc-b564-fe4930e934ad-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.241267 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.241246 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09074706-f21e-40dc-b564-fe4930e934ad-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.242957 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.242928 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09074706-f21e-40dc-b564-fe4930e934ad-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.243262 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.243245 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09074706-f21e-40dc-b564-fe4930e934ad-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.249893 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.249862 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fsqz\" (UniqueName: \"kubernetes.io/projected/09074706-f21e-40dc-b564-fe4930e934ad-kube-api-access-2fsqz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xxqq2\" (UID: \"09074706-f21e-40dc-b564-fe4930e934ad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.347018 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.346912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:14.619572 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.619546 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2"] Sep 05 19:56:14.622050 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:56:14.622020 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09074706_f21e_40dc_b564_fe4930e934ad.slice/crio-8d99b26038ce83f1a0acf2058c842efebaf3f8a00da9e23eb14677ab196d866e WatchSource:0}: Error finding container 8d99b26038ce83f1a0acf2058c842efebaf3f8a00da9e23eb14677ab196d866e: Status 404 returned error can't find the container with id 8d99b26038ce83f1a0acf2058c842efebaf3f8a00da9e23eb14677ab196d866e Sep 05 19:56:14.623434 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:14.623407 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 19:56:15.336780 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:15.336744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" event={"ID":"09074706-f21e-40dc-b564-fe4930e934ad","Type":"ContainerStarted","Data":"8d99b26038ce83f1a0acf2058c842efebaf3f8a00da9e23eb14677ab196d866e"} Sep 05 19:56:16.342728 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:16.342687 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" event={"ID":"09074706-f21e-40dc-b564-fe4930e934ad","Type":"ContainerStarted","Data":"38dbea3dc068e5aff00790453ba73b8cad6f34d1d5e6ecc9b2f4cbaf477bb76e"} Sep 05 19:56:16.362015 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:16.361959 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" podStartSLOduration=1.3349239179999999 podStartE2EDuration="2.36194205s" totalImagesPullingTime="1.027018132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:56:16.361005224 +0000 UTC m=+796.846306108" watchObservedRunningTime="2026-09-05 19:56:16.36194205 +0000 UTC m=+796.847242923" Sep 05 19:56:24.347587 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:24.347493 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:24.347587 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:24.347554 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:24.360464 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:24.360438 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:24.393454 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:24.393423 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xxqq2" Sep 05 19:56:37.321725 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.321689 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv"] Sep 05 19:56:37.327997 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.327974 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.330756 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.330733 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 05 19:56:37.334966 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.334944 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv"] Sep 05 19:56:37.426739 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.426702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19c6907e-797d-4cf7-9519-1434ec53b2ce-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.426739 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.426743 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19c6907e-797d-4cf7-9519-1434ec53b2ce-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.426967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.426849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19c6907e-797d-4cf7-9519-1434ec53b2ce-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.426967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.426865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c6907e-797d-4cf7-9519-1434ec53b2ce-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.426967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.426917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhqft\" (UniqueName: \"kubernetes.io/projected/19c6907e-797d-4cf7-9519-1434ec53b2ce-kube-api-access-vhqft\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.426967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.426956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19c6907e-797d-4cf7-9519-1434ec53b2ce-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.527521 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.527483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19c6907e-797d-4cf7-9519-1434ec53b2ce-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.527521 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.527519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c6907e-797d-4cf7-9519-1434ec53b2ce-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.527756 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.527562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhqft\" (UniqueName: \"kubernetes.io/projected/19c6907e-797d-4cf7-9519-1434ec53b2ce-kube-api-access-vhqft\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.527756 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.527596 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19c6907e-797d-4cf7-9519-1434ec53b2ce-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.527756 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.527645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19c6907e-797d-4cf7-9519-1434ec53b2ce-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.527756 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.527668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19c6907e-797d-4cf7-9519-1434ec53b2ce-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.528072 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.528045 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19c6907e-797d-4cf7-9519-1434ec53b2ce-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.528134 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.528109 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19c6907e-797d-4cf7-9519-1434ec53b2ce-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.528185 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.528145 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19c6907e-797d-4cf7-9519-1434ec53b2ce-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.529891 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.529869 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19c6907e-797d-4cf7-9519-1434ec53b2ce-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.530002 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.529986 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c6907e-797d-4cf7-9519-1434ec53b2ce-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.537007 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.536980 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhqft\" (UniqueName: \"kubernetes.io/projected/19c6907e-797d-4cf7-9519-1434ec53b2ce-kube-api-access-vhqft\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kc5rv\" (UID: \"19c6907e-797d-4cf7-9519-1434ec53b2ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.639175 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.639082 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:37.912149 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:37.912122 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv"] Sep 05 19:56:37.914195 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:56:37.914164 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c6907e_797d_4cf7_9519_1434ec53b2ce.slice/crio-a24c4a30fbff6b7fe500e54b0f189e8ed26bed311f0fb8abf9ade57157a960e1 WatchSource:0}: Error finding container a24c4a30fbff6b7fe500e54b0f189e8ed26bed311f0fb8abf9ade57157a960e1: Status 404 returned error can't find the container with id a24c4a30fbff6b7fe500e54b0f189e8ed26bed311f0fb8abf9ade57157a960e1 Sep 05 19:56:38.446190 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:38.446151 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" event={"ID":"19c6907e-797d-4cf7-9519-1434ec53b2ce","Type":"ContainerStarted","Data":"8b1cbd7748961c6d1a620d3772f89a4f4efd7944e27f453973aeedc0341b0468"} Sep 05 19:56:38.446593 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:38.446204 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" event={"ID":"19c6907e-797d-4cf7-9519-1434ec53b2ce","Type":"ContainerStarted","Data":"a24c4a30fbff6b7fe500e54b0f189e8ed26bed311f0fb8abf9ade57157a960e1"} Sep 05 19:56:38.464942 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:38.464886 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" podStartSLOduration=1.295918022 podStartE2EDuration="1.464868751s" totalImagesPullingTime="168.950729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:56:38.464077377 +0000 UTC m=+818.949378252" watchObservedRunningTime="2026-09-05 19:56:38.464868751 +0000 UTC m=+818.950169624" Sep 05 19:56:47.640115 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.640079 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:47.640605 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.640134 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:47.652781 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.652748 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:47.826195 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.826161 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t"] Sep 05 19:56:47.829815 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.829776 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:47.832561 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.832540 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 05 19:56:47.840758 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.840729 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t"] Sep 05 19:56:47.908866 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.908767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d3c008-dab1-4880-bbdc-c606698e9938-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:47.908866 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.908814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d3c008-dab1-4880-bbdc-c606698e9938-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:47.909048 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.908941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d3c008-dab1-4880-bbdc-c606698e9938-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:47.909048 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.908968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d3c008-dab1-4880-bbdc-c606698e9938-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:47.909048 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.908990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d3c008-dab1-4880-bbdc-c606698e9938-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:47.909048 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:47.909041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5mc4\" (UniqueName: \"kubernetes.io/projected/d2d3c008-dab1-4880-bbdc-c606698e9938-kube-api-access-s5mc4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.009498 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.009469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5mc4\" (UniqueName: \"kubernetes.io/projected/d2d3c008-dab1-4880-bbdc-c606698e9938-kube-api-access-s5mc4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.009723 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.009509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d3c008-dab1-4880-bbdc-c606698e9938-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.009723 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.009526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d3c008-dab1-4880-bbdc-c606698e9938-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.009723 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.009608 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d3c008-dab1-4880-bbdc-c606698e9938-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.009723 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.009642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d3c008-dab1-4880-bbdc-c606698e9938-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.009723 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.009674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d3c008-dab1-4880-bbdc-c606698e9938-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.010043 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.010018 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d3c008-dab1-4880-bbdc-c606698e9938-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.010108 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.010056 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d3c008-dab1-4880-bbdc-c606698e9938-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.010108 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.010088 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d3c008-dab1-4880-bbdc-c606698e9938-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.012016 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.011997 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d3c008-dab1-4880-bbdc-c606698e9938-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.012381 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.012366 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d3c008-dab1-4880-bbdc-c606698e9938-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.018051 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.018029 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5mc4\" (UniqueName: \"kubernetes.io/projected/d2d3c008-dab1-4880-bbdc-c606698e9938-kube-api-access-s5mc4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t\" (UID: \"d2d3c008-dab1-4880-bbdc-c606698e9938\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.140930 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.140887 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:48.421475 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.421432 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t"] Sep 05 19:56:48.423974 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:56:48.423951 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d3c008_dab1_4880_bbdc_c606698e9938.slice/crio-83703fd64f0dc87375947d21d890a218326af0b97e84570650ef1707fc106eb4 WatchSource:0}: Error finding container 83703fd64f0dc87375947d21d890a218326af0b97e84570650ef1707fc106eb4: Status 404 returned error can't find the container with id 83703fd64f0dc87375947d21d890a218326af0b97e84570650ef1707fc106eb4 Sep 05 19:56:48.494016 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.493920 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" event={"ID":"d2d3c008-dab1-4880-bbdc-c606698e9938","Type":"ContainerStarted","Data":"83703fd64f0dc87375947d21d890a218326af0b97e84570650ef1707fc106eb4"} Sep 05 19:56:48.505536 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:48.505506 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kc5rv" Sep 05 19:56:49.499894 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:49.499852 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" event={"ID":"d2d3c008-dab1-4880-bbdc-c606698e9938","Type":"ContainerStarted","Data":"4434f335144ae64cae542bf838a582be6be35fc8148c2f1e808c5f4962f46daa"} Sep 05 19:56:49.520412 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:49.520355 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" podStartSLOduration=2.363164222 podStartE2EDuration="2.52034115s" totalImagesPullingTime="157.176928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:56:49.518958593 +0000 UTC m=+830.004259467" watchObservedRunningTime="2026-09-05 19:56:49.52034115 +0000 UTC m=+830.005642023" Sep 05 19:56:58.142039 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:58.141991 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:58.142422 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:58.142068 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:58.154473 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:58.154447 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:56:58.557902 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:56:58.557862 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t" Sep 05 19:57:26.231724 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.231683 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq"] Sep 05 19:57:26.234877 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.234859 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.237860 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.237837 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 05 19:57:26.246390 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.246366 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq"] Sep 05 19:57:26.350814 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.350769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.350984 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.350946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.350984 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.350976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.351109 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.351015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.351109 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.351056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49g6j\" (UniqueName: \"kubernetes.io/projected/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-kube-api-access-49g6j\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.351198 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.351133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.451881 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.451830 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.452072 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.451910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.452072 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.451994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.452072 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.452017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.452072 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.452053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.452276 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.452076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49g6j\" (UniqueName: \"kubernetes.io/projected/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-kube-api-access-49g6j\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.452276 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.452251 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.452395 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.452368 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.452445 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.452399 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.454244 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.454218 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.454519 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.454502 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.461118 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.461090 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49g6j\" (UniqueName: \"kubernetes.io/projected/fc5c8ede-535a-459c-9a55-bbfb28fe51fd-kube-api-access-49g6j\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6srkq\" (UID: \"fc5c8ede-535a-459c-9a55-bbfb28fe51fd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.544227 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.544192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:26.822787 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:26.822763 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq"] Sep 05 19:57:26.824599 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:57:26.824571 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc5c8ede_535a_459c_9a55_bbfb28fe51fd.slice/crio-af8031ea09cfe7096ab32ac297f1972a35ddddcb7b39ccaf6f1c50496cee6fd0 WatchSource:0}: Error finding container af8031ea09cfe7096ab32ac297f1972a35ddddcb7b39ccaf6f1c50496cee6fd0: Status 404 returned error can't find the container with id af8031ea09cfe7096ab32ac297f1972a35ddddcb7b39ccaf6f1c50496cee6fd0 Sep 05 19:57:27.684526 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:27.684483 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" event={"ID":"fc5c8ede-535a-459c-9a55-bbfb28fe51fd","Type":"ContainerStarted","Data":"7fc1ea09e6ea499b338e6e7d21e4e5fe77b4945dfa83fb75b7d10fcc7aeefbae"} Sep 05 19:57:27.684526 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:27.684532 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" event={"ID":"fc5c8ede-535a-459c-9a55-bbfb28fe51fd","Type":"ContainerStarted","Data":"af8031ea09cfe7096ab32ac297f1972a35ddddcb7b39ccaf6f1c50496cee6fd0"} Sep 05 19:57:27.703279 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:27.703226 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" podStartSLOduration=1.534040841 podStartE2EDuration="1.703207024s" totalImagesPullingTime="169.166183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:57:27.702869811 +0000 UTC m=+868.188170684" watchObservedRunningTime="2026-09-05 19:57:27.703207024 +0000 UTC m=+868.188507899" Sep 05 19:57:36.544434 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:36.544388 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:36.544858 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:36.544448 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:36.557004 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:36.556975 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:57:36.742217 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:57:36.742187 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6srkq" Sep 05 19:58:00.003174 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:58:00.003095 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 19:58:00.005732 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:58:00.005710 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 19:59:00.980189 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:00.980150 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t"] Sep 05 19:59:00.984150 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:00.984124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:00.987077 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:00.987048 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-588081-serving-cert\"" Sep 05 19:59:00.995474 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:00.995449 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t"] Sep 05 19:59:01.041655 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.041614 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8p4x\" (UniqueName: \"kubernetes.io/projected/0a15528c-00ca-4405-8e46-72e5b206c3b6-kube-api-access-k8p4x\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.041655 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.041657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a15528c-00ca-4405-8e46-72e5b206c3b6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.041890 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.041766 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-maas-api-tls\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.041890 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.041822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-metrics-certs\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.142602 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.142557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8p4x\" (UniqueName: \"kubernetes.io/projected/0a15528c-00ca-4405-8e46-72e5b206c3b6-kube-api-access-k8p4x\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.142602 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.142601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a15528c-00ca-4405-8e46-72e5b206c3b6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.142909 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.142657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-maas-api-tls\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.142909 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.142684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-metrics-certs\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.142909 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:59:01.142818 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-serving-cert: secret "maas-api-e2e-inf-a-588081-serving-cert" not found Sep 05 19:59:01.142909 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:59:01.142907 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-maas-api-tls podName:0a15528c-00ca-4405-8e46-72e5b206c3b6 nodeName:}" failed. No retries permitted until 2026-09-05 19:59:01.64288572 +0000 UTC m=+962.128186586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-maas-api-tls") pod "maas-api-e2e-inf-a-588081-578dfb4d48-x472t" (UID: "0a15528c-00ca-4405-8e46-72e5b206c3b6") : secret "maas-api-e2e-inf-a-588081-serving-cert" not found Sep 05 19:59:01.143331 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.143309 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a15528c-00ca-4405-8e46-72e5b206c3b6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.145291 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.145269 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-metrics-certs\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.154513 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.154482 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8p4x\" (UniqueName: \"kubernetes.io/projected/0a15528c-00ca-4405-8e46-72e5b206c3b6-kube-api-access-k8p4x\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.648906 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.648865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-maas-api-tls\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.651213 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.651190 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-maas-api-tls\") pod \"maas-api-e2e-inf-a-588081-578dfb4d48-x472t\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:01.895759 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:01.895713 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:02.172032 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:02.172002 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t"] Sep 05 19:59:02.174781 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:59:02.174754 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a15528c_00ca_4405_8e46_72e5b206c3b6.slice/crio-e216e503758f81b82a6b95d4a60afd9207e890678a9d933ac1873c3d8a828887 WatchSource:0}: Error finding container e216e503758f81b82a6b95d4a60afd9207e890678a9d933ac1873c3d8a828887: Status 404 returned error can't find the container with id e216e503758f81b82a6b95d4a60afd9207e890678a9d933ac1873c3d8a828887 Sep 05 19:59:03.149720 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:03.149686 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" event={"ID":"0a15528c-00ca-4405-8e46-72e5b206c3b6","Type":"ContainerStarted","Data":"26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214"} Sep 05 19:59:03.149888 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:03.149736 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" event={"ID":"0a15528c-00ca-4405-8e46-72e5b206c3b6","Type":"ContainerStarted","Data":"e216e503758f81b82a6b95d4a60afd9207e890678a9d933ac1873c3d8a828887"} Sep 05 19:59:03.149888 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:03.149758 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:08.562146 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.562084 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" podStartSLOduration=8.123832458 podStartE2EDuration="8.56206034s" totalImagesPullingTime="438.227882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:03.178766367 +0000 UTC m=+963.664067241" watchObservedRunningTime="2026-09-05 19:59:08.56206034 +0000 UTC m=+969.047361215" Sep 05 19:59:08.562861 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.562839 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5"] Sep 05 19:59:08.566991 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.566970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:08.569818 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.569778 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b4496e2f-serving-cert\"" Sep 05 19:59:08.580948 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.580925 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5"] Sep 05 19:59:08.716470 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.716428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gpjr\" (UniqueName: \"kubernetes.io/projected/d51f71d0-3c3e-406e-8364-da056efcd6d5-kube-api-access-5gpjr\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:08.716470 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.716473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-maas-api-tls\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:08.716677 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.716550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51f71d0-3c3e-406e-8364-da056efcd6d5-openshift-service-ca\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:08.716677 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.716600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-metrics-certs\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:08.817868 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.817731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-metrics-certs\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:08.818046 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.817899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpjr\" (UniqueName: \"kubernetes.io/projected/d51f71d0-3c3e-406e-8364-da056efcd6d5-kube-api-access-5gpjr\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:08.818046 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.817926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-maas-api-tls\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:08.818046 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.817971 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51f71d0-3c3e-406e-8364-da056efcd6d5-openshift-service-ca\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:08.818212 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:59:08.818083 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-serving-cert: secret "maas-api-e2e-ait-b4496e2f-serving-cert" not found Sep 05 19:59:08.818212 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:59:08.818169 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-maas-api-tls podName:d51f71d0-3c3e-406e-8364-da056efcd6d5 nodeName:}" failed. No retries permitted until 2026-09-05 19:59:09.318146647 +0000 UTC m=+969.803447512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-maas-api-tls") pod "maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" (UID: "d51f71d0-3c3e-406e-8364-da056efcd6d5") : secret "maas-api-e2e-ait-b4496e2f-serving-cert" not found Sep 05 19:59:08.818559 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.818540 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51f71d0-3c3e-406e-8364-da056efcd6d5-openshift-service-ca\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:08.820216 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.820195 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-metrics-certs\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:08.828444 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:08.828417 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gpjr\" (UniqueName: \"kubernetes.io/projected/d51f71d0-3c3e-406e-8364-da056efcd6d5-kube-api-access-5gpjr\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:09.159503 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.159427 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 19:59:09.238661 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.238630 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd"] Sep 05 19:59:09.242759 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.242735 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" Sep 05 19:59:09.245704 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.245676 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b4496e2f\"" Sep 05 19:59:09.245854 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.245768 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b4496e2f-dockercfg-bnfr7\"" Sep 05 19:59:09.254451 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.254418 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd"] Sep 05 19:59:09.323236 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.323197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-maas-api-tls\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:09.323236 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.323237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b7bced-07ae-4164-8809-5ce4d76faa8d-plugins-config-volume\") pod \"payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd\" (UID: \"26b7bced-07ae-4164-8809-5ce4d76faa8d\") " pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" Sep 05 19:59:09.323450 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.323297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl2rd\" (UniqueName: \"kubernetes.io/projected/26b7bced-07ae-4164-8809-5ce4d76faa8d-kube-api-access-xl2rd\") pod \"payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd\" (UID: \"26b7bced-07ae-4164-8809-5ce4d76faa8d\") " pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" Sep 05 19:59:09.325562 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.325540 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-maas-api-tls\") pod \"maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:09.424363 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.424264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b7bced-07ae-4164-8809-5ce4d76faa8d-plugins-config-volume\") pod \"payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd\" (UID: \"26b7bced-07ae-4164-8809-5ce4d76faa8d\") " pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" Sep 05 19:59:09.424363 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.424350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl2rd\" (UniqueName: \"kubernetes.io/projected/26b7bced-07ae-4164-8809-5ce4d76faa8d-kube-api-access-xl2rd\") pod \"payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd\" (UID: \"26b7bced-07ae-4164-8809-5ce4d76faa8d\") " pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" Sep 05 19:59:09.425093 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.425071 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b7bced-07ae-4164-8809-5ce4d76faa8d-plugins-config-volume\") pod \"payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd\" (UID: \"26b7bced-07ae-4164-8809-5ce4d76faa8d\") " pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" Sep 05 19:59:09.433228 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.433205 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl2rd\" (UniqueName: \"kubernetes.io/projected/26b7bced-07ae-4164-8809-5ce4d76faa8d-kube-api-access-xl2rd\") pod \"payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd\" (UID: \"26b7bced-07ae-4164-8809-5ce4d76faa8d\") " pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" Sep 05 19:59:09.478195 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.478156 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:09.554352 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.554321 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" Sep 05 19:59:09.752661 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.752624 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5"] Sep 05 19:59:09.755250 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:59:09.755218 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51f71d0_3c3e_406e_8364_da056efcd6d5.slice/crio-5e6e42334d8a9e34448f519632b74c44940d5b87bfc8893de9ba742150f507ed WatchSource:0}: Error finding container 5e6e42334d8a9e34448f519632b74c44940d5b87bfc8893de9ba742150f507ed: Status 404 returned error can't find the container with id 5e6e42334d8a9e34448f519632b74c44940d5b87bfc8893de9ba742150f507ed Sep 05 19:59:09.837743 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:09.837719 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd"] Sep 05 19:59:09.840479 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:59:09.840447 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b7bced_07ae_4164_8809_5ce4d76faa8d.slice/crio-038a17720083695b463bc746ec984759bad4e7769dc63d081bb5178a81542017 WatchSource:0}: Error finding container 038a17720083695b463bc746ec984759bad4e7769dc63d081bb5178a81542017: Status 404 returned error can't find the container with id 038a17720083695b463bc746ec984759bad4e7769dc63d081bb5178a81542017 Sep 05 19:59:10.188324 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:10.188283 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" event={"ID":"d51f71d0-3c3e-406e-8364-da056efcd6d5","Type":"ContainerStarted","Data":"5e6e42334d8a9e34448f519632b74c44940d5b87bfc8893de9ba742150f507ed"} Sep 05 19:59:10.190260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:10.190232 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" event={"ID":"26b7bced-07ae-4164-8809-5ce4d76faa8d","Type":"ContainerStarted","Data":"eca78df904a3f11bad4c316b9077725b3a74ddb2f87e31c25d3febdeafbf4e5a"} Sep 05 19:59:10.190450 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:10.190430 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" event={"ID":"26b7bced-07ae-4164-8809-5ce4d76faa8d","Type":"ContainerStarted","Data":"038a17720083695b463bc746ec984759bad4e7769dc63d081bb5178a81542017"} Sep 05 19:59:10.190561 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:10.190547 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" Sep 05 19:59:10.221695 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:10.221637 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" podStartSLOduration=1.221615705 podStartE2EDuration="1.221615705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:10.221324062 +0000 UTC m=+970.706624936" watchObservedRunningTime="2026-09-05 19:59:10.221615705 +0000 UTC m=+970.706916579" Sep 05 19:59:10.909093 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:10.909051 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2"] Sep 05 19:59:10.912474 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:10.912456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" Sep 05 19:59:10.915303 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:10.915278 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-1fd18b-dockercfg-wmkbr\"" Sep 05 19:59:10.915303 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:10.915295 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-1fd18b\"" Sep 05 19:59:10.926315 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:10.926285 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2"] Sep 05 19:59:11.039902 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:11.039850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2\" (UID: \"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" Sep 05 19:59:11.039902 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:11.039893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qltxj\" (UniqueName: \"kubernetes.io/projected/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5-kube-api-access-qltxj\") pod \"payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2\" (UID: \"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" Sep 05 19:59:11.140904 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:11.140864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2\" (UID: \"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" Sep 05 19:59:11.140904 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:11.140905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qltxj\" (UniqueName: \"kubernetes.io/projected/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5-kube-api-access-qltxj\") pod \"payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2\" (UID: \"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" Sep 05 19:59:11.141624 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:11.141603 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2\" (UID: \"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" Sep 05 19:59:11.149899 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:11.149868 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qltxj\" (UniqueName: \"kubernetes.io/projected/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5-kube-api-access-qltxj\") pod \"payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2\" (UID: \"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" Sep 05 19:59:11.198418 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:11.198327 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" event={"ID":"d51f71d0-3c3e-406e-8364-da056efcd6d5","Type":"ContainerStarted","Data":"a76506aa90e87a2797cc12ae1d2a98153982892140ba3d69e0b7322896769b2b"} Sep 05 19:59:11.198701 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:11.198542 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:11.219780 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:11.219727 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" podStartSLOduration=2.705464475 podStartE2EDuration="3.21970886s" totalImagesPullingTime="514.244385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:11.216336616 +0000 UTC m=+971.701637488" watchObservedRunningTime="2026-09-05 19:59:11.21970886 +0000 UTC m=+971.705009732" Sep 05 19:59:11.222603 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:11.222582 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" Sep 05 19:59:11.497441 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:11.497414 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2"] Sep 05 19:59:11.499587 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:59:11.499564 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21bad62_a2c3_47ea_9a01_3a5b4598e5c5.slice/crio-710f2aab074b1aa2fe5be442594feefe058e59e7603e500a8221d7a6d1430f08 WatchSource:0}: Error finding container 710f2aab074b1aa2fe5be442594feefe058e59e7603e500a8221d7a6d1430f08: Status 404 returned error can't find the container with id 710f2aab074b1aa2fe5be442594feefe058e59e7603e500a8221d7a6d1430f08 Sep 05 19:59:12.204577 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:12.204537 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" event={"ID":"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5","Type":"ContainerStarted","Data":"2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c"} Sep 05 19:59:12.205033 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:12.204604 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" event={"ID":"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5","Type":"ContainerStarted","Data":"710f2aab074b1aa2fe5be442594feefe058e59e7603e500a8221d7a6d1430f08"} Sep 05 19:59:12.205033 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:12.204962 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" Sep 05 19:59:17.210425 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:17.210386 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:17.228443 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:17.228390 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" podStartSLOduration=7.228371797 podStartE2EDuration="7.228371797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:12.222400512 +0000 UTC m=+972.707701386" watchObservedRunningTime="2026-09-05 19:59:17.228371797 +0000 UTC m=+977.713672670" Sep 05 19:59:21.200676 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:21.200643 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" Sep 05 19:59:23.212487 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:23.212404 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" Sep 05 19:59:32.395365 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.395323 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk"] Sep 05 19:59:32.399266 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.399238 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:32.402841 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.402813 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1dc7b1-serving-cert\"" Sep 05 19:59:32.424092 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.424049 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk"] Sep 05 19:59:32.453992 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.453954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nt44\" (UniqueName: \"kubernetes.io/projected/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-kube-api-access-6nt44\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:32.454188 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.453996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-maas-api-tls\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:32.454188 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.454081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:32.454188 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.454150 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-metrics-certs\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:32.554961 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.554920 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:32.555201 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.554974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-metrics-certs\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:32.555201 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.555045 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nt44\" (UniqueName: \"kubernetes.io/projected/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-kube-api-access-6nt44\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:32.555201 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.555063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-maas-api-tls\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:32.555201 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:59:32.555156 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-serving-cert: secret "maas-api-e2e-inf-b-1dc7b1-serving-cert" not found Sep 05 19:59:32.555460 ip-10-0-143-252 kubenswrapper[2578]: E0905 19:59:32.555225 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-maas-api-tls podName:03c1f839-7fd4-4c3f-a65a-915f0689b1ce nodeName:}" failed. No retries permitted until 2026-09-05 19:59:33.05520392 +0000 UTC m=+993.540504771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-maas-api-tls") pod "maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" (UID: "03c1f839-7fd4-4c3f-a65a-915f0689b1ce") : secret "maas-api-e2e-inf-b-1dc7b1-serving-cert" not found Sep 05 19:59:32.555759 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.555733 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:32.557483 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.557453 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-metrics-certs\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:32.567844 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:32.567822 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nt44\" (UniqueName: \"kubernetes.io/projected/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-kube-api-access-6nt44\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:33.061674 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:33.061635 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-maas-api-tls\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:33.064030 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:33.064004 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-maas-api-tls\") pod \"maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:33.309517 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:33.309468 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:33.589122 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:33.588935 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk"] Sep 05 19:59:33.591317 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:59:33.591290 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c1f839_7fd4_4c3f_a65a_915f0689b1ce.slice/crio-58775ce40140ad9cf7c057c281d85ebc136d2b072d78c592a5e7124511fb90e6 WatchSource:0}: Error finding container 58775ce40140ad9cf7c057c281d85ebc136d2b072d78c592a5e7124511fb90e6: Status 404 returned error can't find the container with id 58775ce40140ad9cf7c057c281d85ebc136d2b072d78c592a5e7124511fb90e6 Sep 05 19:59:34.320290 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:34.320250 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" event={"ID":"03c1f839-7fd4-4c3f-a65a-915f0689b1ce","Type":"ContainerStarted","Data":"bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc"} Sep 05 19:59:34.320494 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:34.320303 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" event={"ID":"03c1f839-7fd4-4c3f-a65a-915f0689b1ce","Type":"ContainerStarted","Data":"58775ce40140ad9cf7c057c281d85ebc136d2b072d78c592a5e7124511fb90e6"} Sep 05 19:59:34.320494 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:34.320349 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:34.341014 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:34.340958 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" podStartSLOduration=2.014422515 podStartE2EDuration="2.340943004s" totalImagesPullingTime="326.520489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:34.340072492 +0000 UTC m=+994.825373364" watchObservedRunningTime="2026-09-05 19:59:34.340943004 +0000 UTC m=+994.826243877" Sep 05 19:59:40.331107 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:40.331070 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 19:59:49.223619 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.223584 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5"] Sep 05 19:59:49.224153 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.223846 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" podUID="d51f71d0-3c3e-406e-8364-da056efcd6d5" containerName="maas-api" containerID="cri-o://a76506aa90e87a2797cc12ae1d2a98153982892140ba3d69e0b7322896769b2b" gracePeriod=30 Sep 05 19:59:49.281106 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.281066 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd"] Sep 05 19:59:49.282162 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.281812 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" podUID="26b7bced-07ae-4164-8809-5ce4d76faa8d" containerName="payload-processing" containerID="cri-o://eca78df904a3f11bad4c316b9077725b3a74ddb2f87e31c25d3febdeafbf4e5a" gracePeriod=30 Sep 05 19:59:49.398445 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.398409 2578 generic.go:356] "Generic (PLEG): container finished" podID="26b7bced-07ae-4164-8809-5ce4d76faa8d" containerID="eca78df904a3f11bad4c316b9077725b3a74ddb2f87e31c25d3febdeafbf4e5a" exitCode=0 Sep 05 19:59:49.398579 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.398490 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" event={"ID":"26b7bced-07ae-4164-8809-5ce4d76faa8d","Type":"ContainerDied","Data":"eca78df904a3f11bad4c316b9077725b3a74ddb2f87e31c25d3febdeafbf4e5a"} Sep 05 19:59:49.400650 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.400619 2578 generic.go:356] "Generic (PLEG): container finished" podID="d51f71d0-3c3e-406e-8364-da056efcd6d5" containerID="a76506aa90e87a2797cc12ae1d2a98153982892140ba3d69e0b7322896769b2b" exitCode=0 Sep 05 19:59:49.400768 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.400705 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" event={"ID":"d51f71d0-3c3e-406e-8364-da056efcd6d5","Type":"ContainerDied","Data":"a76506aa90e87a2797cc12ae1d2a98153982892140ba3d69e0b7322896769b2b"} Sep 05 19:59:49.549456 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.549427 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:49.552613 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.552591 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" Sep 05 19:59:49.725682 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.725650 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-maas-api-tls\") pod \"d51f71d0-3c3e-406e-8364-da056efcd6d5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " Sep 05 19:59:49.725880 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.725720 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51f71d0-3c3e-406e-8364-da056efcd6d5-openshift-service-ca\") pod \"d51f71d0-3c3e-406e-8364-da056efcd6d5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " Sep 05 19:59:49.725880 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.725839 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl2rd\" (UniqueName: \"kubernetes.io/projected/26b7bced-07ae-4164-8809-5ce4d76faa8d-kube-api-access-xl2rd\") pod \"26b7bced-07ae-4164-8809-5ce4d76faa8d\" (UID: \"26b7bced-07ae-4164-8809-5ce4d76faa8d\") " Sep 05 19:59:49.725970 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.725895 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b7bced-07ae-4164-8809-5ce4d76faa8d-plugins-config-volume\") pod \"26b7bced-07ae-4164-8809-5ce4d76faa8d\" (UID: \"26b7bced-07ae-4164-8809-5ce4d76faa8d\") " Sep 05 19:59:49.725970 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.725937 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gpjr\" (UniqueName: \"kubernetes.io/projected/d51f71d0-3c3e-406e-8364-da056efcd6d5-kube-api-access-5gpjr\") pod \"d51f71d0-3c3e-406e-8364-da056efcd6d5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " Sep 05 19:59:49.726058 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.725974 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-metrics-certs\") pod \"d51f71d0-3c3e-406e-8364-da056efcd6d5\" (UID: \"d51f71d0-3c3e-406e-8364-da056efcd6d5\") " Sep 05 19:59:49.726109 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.726086 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d51f71d0-3c3e-406e-8364-da056efcd6d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d51f71d0-3c3e-406e-8364-da056efcd6d5" (UID: "d51f71d0-3c3e-406e-8364-da056efcd6d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:59:49.726320 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.726298 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b7bced-07ae-4164-8809-5ce4d76faa8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26b7bced-07ae-4164-8809-5ce4d76faa8d" (UID: "26b7bced-07ae-4164-8809-5ce4d76faa8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 19:59:49.726427 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.726386 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b7bced-07ae-4164-8809-5ce4d76faa8d-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:59:49.726427 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.726403 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51f71d0-3c3e-406e-8364-da056efcd6d5-openshift-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:59:49.728200 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.728133 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d51f71d0-3c3e-406e-8364-da056efcd6d5" (UID: "d51f71d0-3c3e-406e-8364-da056efcd6d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:59:49.728200 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.728173 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b7bced-07ae-4164-8809-5ce4d76faa8d-kube-api-access-xl2rd" (OuterVolumeSpecName: "kube-api-access-xl2rd") pod "26b7bced-07ae-4164-8809-5ce4d76faa8d" (UID: "26b7bced-07ae-4164-8809-5ce4d76faa8d"). InnerVolumeSpecName "kube-api-access-xl2rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:59:49.728356 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.728337 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d51f71d0-3c3e-406e-8364-da056efcd6d5" (UID: "d51f71d0-3c3e-406e-8364-da056efcd6d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 19:59:49.728419 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.728337 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d51f71d0-3c3e-406e-8364-da056efcd6d5-kube-api-access-5gpjr" (OuterVolumeSpecName: "kube-api-access-5gpjr") pod "d51f71d0-3c3e-406e-8364-da056efcd6d5" (UID: "d51f71d0-3c3e-406e-8364-da056efcd6d5"). InnerVolumeSpecName "kube-api-access-5gpjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 19:59:49.827242 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.827203 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gpjr\" (UniqueName: \"kubernetes.io/projected/d51f71d0-3c3e-406e-8364-da056efcd6d5-kube-api-access-5gpjr\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:59:49.827242 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.827235 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:59:49.827242 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.827245 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d51f71d0-3c3e-406e-8364-da056efcd6d5-maas-api-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:59:49.827242 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:49.827253 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl2rd\" (UniqueName: \"kubernetes.io/projected/26b7bced-07ae-4164-8809-5ce4d76faa8d-kube-api-access-xl2rd\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 19:59:50.406768 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.406681 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" Sep 05 19:59:50.406768 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.406707 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5" event={"ID":"d51f71d0-3c3e-406e-8364-da056efcd6d5","Type":"ContainerDied","Data":"5e6e42334d8a9e34448f519632b74c44940d5b87bfc8893de9ba742150f507ed"} Sep 05 19:59:50.407280 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.406834 2578 scope.go:117] "RemoveContainer" containerID="a76506aa90e87a2797cc12ae1d2a98153982892140ba3d69e0b7322896769b2b" Sep 05 19:59:50.408268 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.408099 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" event={"ID":"26b7bced-07ae-4164-8809-5ce4d76faa8d","Type":"ContainerDied","Data":"038a17720083695b463bc746ec984759bad4e7769dc63d081bb5178a81542017"} Sep 05 19:59:50.408268 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.408188 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd" Sep 05 19:59:50.416407 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.416385 2578 scope.go:117] "RemoveContainer" containerID="eca78df904a3f11bad4c316b9077725b3a74ddb2f87e31c25d3febdeafbf4e5a" Sep 05 19:59:50.430947 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.430921 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5"] Sep 05 19:59:50.435129 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.435108 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4496e2f-8968d9945-8m8z5"] Sep 05 19:59:50.452527 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.452494 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd"] Sep 05 19:59:50.454402 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.454379 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b4496e2f-78c648df5-8mjdd"] Sep 05 19:59:50.485057 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.485027 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc"] Sep 05 19:59:50.485399 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.485388 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d51f71d0-3c3e-406e-8364-da056efcd6d5" containerName="maas-api" Sep 05 19:59:50.485438 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.485400 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d51f71d0-3c3e-406e-8364-da056efcd6d5" containerName="maas-api" Sep 05 19:59:50.485438 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.485418 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b7bced-07ae-4164-8809-5ce4d76faa8d" containerName="payload-processing" Sep 05 19:59:50.485438 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.485424 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b7bced-07ae-4164-8809-5ce4d76faa8d" containerName="payload-processing" Sep 05 19:59:50.485531 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.485485 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b7bced-07ae-4164-8809-5ce4d76faa8d" containerName="payload-processing" Sep 05 19:59:50.485531 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.485494 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d51f71d0-3c3e-406e-8364-da056efcd6d5" containerName="maas-api" Sep 05 19:59:50.490257 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.490232 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.493073 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.493049 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 05 19:59:50.493210 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.493049 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e1fbf0a6-gw-openshift-default-dockercfg-hmz7s\"" Sep 05 19:59:50.503468 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.503441 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc"] Sep 05 19:59:50.635922 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.635883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-workload-certs\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.635922 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.635922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592a6634-6158-4697-b423-0407a3748b56-istio-token\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.636124 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.635950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tt6f\" (UniqueName: \"kubernetes.io/projected/592a6634-6158-4697-b423-0407a3748b56-kube-api-access-4tt6f\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.636124 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.636031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-istio-data\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.636124 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.636069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-istio-envoy\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.636124 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.636100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-credential-socket\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.636253 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.636146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-workload-socket\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.636253 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.636186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592a6634-6158-4697-b423-0407a3748b56-istiod-ca-cert\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.636253 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.636224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592a6634-6158-4697-b423-0407a3748b56-istio-podinfo\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.736977 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.736880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592a6634-6158-4697-b423-0407a3748b56-istiod-ca-cert\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.736977 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.736946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592a6634-6158-4697-b423-0407a3748b56-istio-podinfo\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.737204 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.736995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-workload-certs\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.737204 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.737020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592a6634-6158-4697-b423-0407a3748b56-istio-token\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.737204 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.737057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tt6f\" (UniqueName: \"kubernetes.io/projected/592a6634-6158-4697-b423-0407a3748b56-kube-api-access-4tt6f\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.737204 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.737090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-istio-data\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.737204 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.737124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-istio-envoy\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.737445 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.737236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-credential-socket\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.737445 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.737300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-workload-socket\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.737445 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.737401 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-workload-certs\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.737615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.737590 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-istio-data\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.737765 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.737731 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592a6634-6158-4697-b423-0407a3748b56-istiod-ca-cert\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.737765 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.737761 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-credential-socket\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.737978 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.737872 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-workload-socket\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.739484 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.739459 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-istio-envoy\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.739866 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.739845 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592a6634-6158-4697-b423-0407a3748b56-istio-podinfo\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.746268 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.746246 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592a6634-6158-4697-b423-0407a3748b56-istio-token\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.746611 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.746594 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tt6f\" (UniqueName: \"kubernetes.io/projected/592a6634-6158-4697-b423-0407a3748b56-kube-api-access-4tt6f\") pod \"e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:50.803342 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:50.803299 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:51.086586 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:51.086552 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc"] Sep 05 19:59:51.088330 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:59:51.088305 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592a6634_6158_4697_b423_0407a3748b56.slice/crio-cf969f17e98a7c3c9e2c1b231357f621e9d3cf6aa764176da562c566553b5834 WatchSource:0}: Error finding container cf969f17e98a7c3c9e2c1b231357f621e9d3cf6aa764176da562c566553b5834: Status 404 returned error can't find the container with id cf969f17e98a7c3c9e2c1b231357f621e9d3cf6aa764176da562c566553b5834 Sep 05 19:59:51.416105 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:51.416020 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" event={"ID":"592a6634-6158-4697-b423-0407a3748b56","Type":"ContainerStarted","Data":"cf969f17e98a7c3c9e2c1b231357f621e9d3cf6aa764176da562c566553b5834"} Sep 05 19:59:52.015260 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:52.015229 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26b7bced-07ae-4164-8809-5ce4d76faa8d" path="/var/lib/kubelet/pods/26b7bced-07ae-4164-8809-5ce4d76faa8d/volumes" Sep 05 19:59:52.015807 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:52.015776 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d51f71d0-3c3e-406e-8364-da056efcd6d5" path="/var/lib/kubelet/pods/d51f71d0-3c3e-406e-8364-da056efcd6d5/volumes" Sep 05 19:59:53.393695 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:53.393657 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 05 19:59:53.393986 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:53.393732 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 05 19:59:53.393986 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:53.393761 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 05 19:59:54.433972 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:54.433936 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" event={"ID":"592a6634-6158-4697-b423-0407a3748b56","Type":"ContainerStarted","Data":"4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d"} Sep 05 19:59:54.456550 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:54.456496 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" podStartSLOduration=2.153796873 podStartE2EDuration="4.45648046s" totalImagesPullingTime="2.302683587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:54.454094679 +0000 UTC m=+1014.939395546" watchObservedRunningTime="2026-09-05 19:59:54.45648046 +0000 UTC m=+1014.941781332" Sep 05 19:59:54.803872 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:54.803832 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:54.809405 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:54.809374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:55.439669 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:55.439633 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:55.440615 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:55.440592 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 19:59:56.625188 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:56.625149 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv"] Sep 05 19:59:56.629025 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:56.628999 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" Sep 05 19:59:56.631986 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:56.631963 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e1fbf0a6\"" Sep 05 19:59:56.632140 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:56.632120 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e1fbf0a6-dockercfg-b4kq9\"" Sep 05 19:59:56.640700 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:56.640678 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv"] Sep 05 19:59:56.796902 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:56.796850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3dea27-dd70-4406-a879-2ba9d72e61c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv\" (UID: \"0c3dea27-dd70-4406-a879-2ba9d72e61c6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" Sep 05 19:59:56.797093 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:56.797036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzkg7\" (UniqueName: \"kubernetes.io/projected/0c3dea27-dd70-4406-a879-2ba9d72e61c6-kube-api-access-kzkg7\") pod \"payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv\" (UID: \"0c3dea27-dd70-4406-a879-2ba9d72e61c6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" Sep 05 19:59:56.897625 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:56.897523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzkg7\" (UniqueName: \"kubernetes.io/projected/0c3dea27-dd70-4406-a879-2ba9d72e61c6-kube-api-access-kzkg7\") pod \"payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv\" (UID: \"0c3dea27-dd70-4406-a879-2ba9d72e61c6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" Sep 05 19:59:56.897625 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:56.897583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3dea27-dd70-4406-a879-2ba9d72e61c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv\" (UID: \"0c3dea27-dd70-4406-a879-2ba9d72e61c6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" Sep 05 19:59:56.898355 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:56.898332 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3dea27-dd70-4406-a879-2ba9d72e61c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv\" (UID: \"0c3dea27-dd70-4406-a879-2ba9d72e61c6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" Sep 05 19:59:56.907052 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:56.907024 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzkg7\" (UniqueName: \"kubernetes.io/projected/0c3dea27-dd70-4406-a879-2ba9d72e61c6-kube-api-access-kzkg7\") pod \"payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv\" (UID: \"0c3dea27-dd70-4406-a879-2ba9d72e61c6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" Sep 05 19:59:56.939865 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:56.939826 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" Sep 05 19:59:57.214967 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:57.214940 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv"] Sep 05 19:59:57.217125 ip-10-0-143-252 kubenswrapper[2578]: W0905 19:59:57.217091 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3dea27_dd70_4406_a879_2ba9d72e61c6.slice/crio-4d17b3f5706c81d3374724f7c62800910c0c63238e278556ec3fe8e1a5a32550 WatchSource:0}: Error finding container 4d17b3f5706c81d3374724f7c62800910c0c63238e278556ec3fe8e1a5a32550: Status 404 returned error can't find the container with id 4d17b3f5706c81d3374724f7c62800910c0c63238e278556ec3fe8e1a5a32550 Sep 05 19:59:57.452969 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:57.452925 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" event={"ID":"0c3dea27-dd70-4406-a879-2ba9d72e61c6","Type":"ContainerStarted","Data":"0be571ecf848ebe79696dee3374378b0889ab2d671b467c5bb4125871e8297f5"} Sep 05 19:59:57.453157 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:57.452994 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" event={"ID":"0c3dea27-dd70-4406-a879-2ba9d72e61c6","Type":"ContainerStarted","Data":"4d17b3f5706c81d3374724f7c62800910c0c63238e278556ec3fe8e1a5a32550"} Sep 05 19:59:58.458279 ip-10-0-143-252 kubenswrapper[2578]: I0905 19:59:58.458249 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" Sep 05 20:00:09.465885 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:09.465847 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" Sep 05 20:00:09.487652 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:09.487594 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" podStartSLOduration=13.487578915 podStartE2EDuration="13.487578915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:57.470343978 +0000 UTC m=+1017.955644852" watchObservedRunningTime="2026-09-05 20:00:09.487578915 +0000 UTC m=+1029.972879789" Sep 05 20:00:10.761187 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:10.760228 2578 patch_prober.go:28] interesting pod/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": read tcp 10.132.0.2:60976->10.132.0.45:15021: read: connection reset by peer" start-of-body= Sep 05 20:00:10.761187 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:10.760316 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": read tcp 10.132.0.2:60976->10.132.0.45:15021: read: connection reset by peer" Sep 05 20:00:11.529073 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:11.529039 2578 generic.go:356] "Generic (PLEG): container finished" podID="592a6634-6158-4697-b423-0407a3748b56" containerID="4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d" exitCode=0 Sep 05 20:00:11.529249 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:11.529091 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" event={"ID":"592a6634-6158-4697-b423-0407a3748b56","Type":"ContainerDied","Data":"4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d"} Sep 05 20:00:11.529462 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:11.529450 2578 scope.go:117] "RemoveContainer" containerID="4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d" Sep 05 20:00:11.530609 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:11.530583 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 05 20:00:11.530665 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:11.530651 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 05 20:00:11.530700 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:11.530684 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 05 20:00:12.536414 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:12.536376 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" event={"ID":"592a6634-6158-4697-b423-0407a3748b56","Type":"ContainerStarted","Data":"77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563"} Sep 05 20:00:12.805002 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:12.804421 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 20:00:12.805002 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:12.804552 2578 patch_prober.go:28] interesting pod/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 05 20:00:12.805002 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:12.804608 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 05 20:00:13.804656 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:13.804608 2578 patch_prober.go:28] interesting pod/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 05 20:00:13.805171 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:13.804700 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 05 20:00:14.804236 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:14.804197 2578 patch_prober.go:28] interesting pod/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 05 20:00:14.804520 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:14.804491 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 05 20:00:15.804789 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:15.804748 2578 patch_prober.go:28] interesting pod/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 05 20:00:15.805411 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:15.805378 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 05 20:00:16.804392 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:16.804346 2578 patch_prober.go:28] interesting pod/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 05 20:00:16.804607 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:16.804451 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 05 20:00:17.834139 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:17.834104 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 20:00:17.834622 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:17.834590 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 20:00:17.835910 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:17.835886 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 20:00:27.401249 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.401194 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv"] Sep 05 20:00:27.402257 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.402192 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" podUID="0c3dea27-dd70-4406-a879-2ba9d72e61c6" containerName="payload-pre-processing" containerID="cri-o://0be571ecf848ebe79696dee3374378b0889ab2d671b467c5bb4125871e8297f5" gracePeriod=30 Sep 05 20:00:27.620626 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.620589 2578 generic.go:356] "Generic (PLEG): container finished" podID="0c3dea27-dd70-4406-a879-2ba9d72e61c6" containerID="0be571ecf848ebe79696dee3374378b0889ab2d671b467c5bb4125871e8297f5" exitCode=0 Sep 05 20:00:27.620844 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.620660 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" event={"ID":"0c3dea27-dd70-4406-a879-2ba9d72e61c6","Type":"ContainerDied","Data":"0be571ecf848ebe79696dee3374378b0889ab2d671b467c5bb4125871e8297f5"} Sep 05 20:00:27.655660 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.655591 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" Sep 05 20:00:27.687588 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.687555 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc"] Sep 05 20:00:27.687804 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.687767 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" containerID="cri-o://77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563" gracePeriod=30 Sep 05 20:00:27.782986 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.782944 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3dea27-dd70-4406-a879-2ba9d72e61c6-plugins-config-volume\") pod \"0c3dea27-dd70-4406-a879-2ba9d72e61c6\" (UID: \"0c3dea27-dd70-4406-a879-2ba9d72e61c6\") " Sep 05 20:00:27.782986 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.782992 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzkg7\" (UniqueName: \"kubernetes.io/projected/0c3dea27-dd70-4406-a879-2ba9d72e61c6-kube-api-access-kzkg7\") pod \"0c3dea27-dd70-4406-a879-2ba9d72e61c6\" (UID: \"0c3dea27-dd70-4406-a879-2ba9d72e61c6\") " Sep 05 20:00:27.783462 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.783424 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3dea27-dd70-4406-a879-2ba9d72e61c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c3dea27-dd70-4406-a879-2ba9d72e61c6" (UID: "0c3dea27-dd70-4406-a879-2ba9d72e61c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:00:27.785220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.785189 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3dea27-dd70-4406-a879-2ba9d72e61c6-kube-api-access-kzkg7" (OuterVolumeSpecName: "kube-api-access-kzkg7") pod "0c3dea27-dd70-4406-a879-2ba9d72e61c6" (UID: "0c3dea27-dd70-4406-a879-2ba9d72e61c6"). InnerVolumeSpecName "kube-api-access-kzkg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:00:27.884214 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.884176 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3dea27-dd70-4406-a879-2ba9d72e61c6-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:00:27.884214 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:27.884208 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzkg7\" (UniqueName: \"kubernetes.io/projected/0c3dea27-dd70-4406-a879-2ba9d72e61c6-kube-api-access-kzkg7\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:00:28.626885 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:28.626844 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" event={"ID":"0c3dea27-dd70-4406-a879-2ba9d72e61c6","Type":"ContainerDied","Data":"4d17b3f5706c81d3374724f7c62800910c0c63238e278556ec3fe8e1a5a32550"} Sep 05 20:00:28.626885 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:28.626875 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv" Sep 05 20:00:28.627357 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:28.626925 2578 scope.go:117] "RemoveContainer" containerID="0be571ecf848ebe79696dee3374378b0889ab2d671b467c5bb4125871e8297f5" Sep 05 20:00:28.644758 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:28.644729 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv"] Sep 05 20:00:28.648887 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:28.648862 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e1fbf0a6-68bd8656db-449vv"] Sep 05 20:00:29.521612 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.521569 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk"] Sep 05 20:00:29.522008 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.521993 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c3dea27-dd70-4406-a879-2ba9d72e61c6" containerName="payload-pre-processing" Sep 05 20:00:29.522063 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.522010 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c3dea27-dd70-4406-a879-2ba9d72e61c6" containerName="payload-pre-processing" Sep 05 20:00:29.522103 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.522075 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c3dea27-dd70-4406-a879-2ba9d72e61c6" containerName="payload-pre-processing" Sep 05 20:00:29.526169 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.526152 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.530493 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.530468 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1fd18b\"/\"default-dockercfg-5ntrf\"" Sep 05 20:00:29.530584 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.530522 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1fd18b\"/\"autores-model-1fd18b-kserve-self-signed-certs\"" Sep 05 20:00:29.530584 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.530575 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1fd18b\"/\"openshift-service-ca.crt\"" Sep 05 20:00:29.530663 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.530528 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1fd18b\"/\"kube-root-ca.crt\"" Sep 05 20:00:29.536852 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.536827 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk"] Sep 05 20:00:29.602500 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.602463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-dshm\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.602660 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.602540 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c47bd2-9436-471b-b03e-77e1c32256da-tls-certs\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.602660 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.602566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-home\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.602660 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.602588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd7bl\" (UniqueName: \"kubernetes.io/projected/76c47bd2-9436-471b-b03e-77e1c32256da-kube-api-access-zd7bl\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.602765 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.602683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-model-cache\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.602765 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.602757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-tmp-dir\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.704067 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.704027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c47bd2-9436-471b-b03e-77e1c32256da-tls-certs\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.704067 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.704068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-home\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.704548 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.704089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd7bl\" (UniqueName: \"kubernetes.io/projected/76c47bd2-9436-471b-b03e-77e1c32256da-kube-api-access-zd7bl\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.704548 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.704121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-model-cache\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.704548 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.704191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-tmp-dir\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.704548 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.704237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-dshm\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.704548 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.704510 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-home\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.704733 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.704598 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-tmp-dir\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.704782 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.704746 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-model-cache\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.706413 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.706393 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-dshm\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.706697 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.706677 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c47bd2-9436-471b-b03e-77e1c32256da-tls-certs\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.712371 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.712348 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd7bl\" (UniqueName: \"kubernetes.io/projected/76c47bd2-9436-471b-b03e-77e1c32256da-kube-api-access-zd7bl\") pod \"autores-model-1fd18b-kserve-75b7dfcc95-tvmbk\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:29.835714 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:29.835616 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:30.013069 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:30.013026 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c3dea27-dd70-4406-a879-2ba9d72e61c6" path="/var/lib/kubelet/pods/0c3dea27-dd70-4406-a879-2ba9d72e61c6/volumes" Sep 05 20:00:30.109482 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:30.109457 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk"] Sep 05 20:00:30.112057 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:00:30.112028 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76c47bd2_9436_471b_b03e_77e1c32256da.slice/crio-2b065a4a13efdfc01810dc55d08733d08f7fd17e5adacd3194e34c1cb9ae0fff WatchSource:0}: Error finding container 2b065a4a13efdfc01810dc55d08733d08f7fd17e5adacd3194e34c1cb9ae0fff: Status 404 returned error can't find the container with id 2b065a4a13efdfc01810dc55d08733d08f7fd17e5adacd3194e34c1cb9ae0fff Sep 05 20:00:30.640073 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:30.640035 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" event={"ID":"76c47bd2-9436-471b-b03e-77e1c32256da","Type":"ContainerStarted","Data":"42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706"} Sep 05 20:00:30.640254 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:30.640081 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" event={"ID":"76c47bd2-9436-471b-b03e-77e1c32256da","Type":"ContainerStarted","Data":"2b065a4a13efdfc01810dc55d08733d08f7fd17e5adacd3194e34c1cb9ae0fff"} Sep 05 20:00:30.658693 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:30.658643 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" podStartSLOduration=1.658627123 podStartE2EDuration="1.658627123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:00:30.656944162 +0000 UTC m=+1051.142245036" watchObservedRunningTime="2026-09-05 20:00:30.658627123 +0000 UTC m=+1051.143927996" Sep 05 20:00:32.836295 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:32.836258 2578 patch_prober.go:28] interesting pod/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 05 20:00:32.836625 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:32.836318 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 05 20:00:32.883536 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:32.883501 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm"] Sep 05 20:00:32.887099 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:32.887074 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" Sep 05 20:00:32.890465 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:32.890441 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a7428d13\"" Sep 05 20:00:32.890465 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:32.890455 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a7428d13-dockercfg-rcqdf\"" Sep 05 20:00:32.898601 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:32.898570 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm"] Sep 05 20:00:32.953585 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:32.953558 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 20:00:33.036928 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.036885 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592a6634-6158-4697-b423-0407a3748b56-istio-token\") pod \"592a6634-6158-4697-b423-0407a3748b56\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " Sep 05 20:00:33.036928 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.036930 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-istio-data\") pod \"592a6634-6158-4697-b423-0407a3748b56\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " Sep 05 20:00:33.037182 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.036971 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-workload-certs\") pod \"592a6634-6158-4697-b423-0407a3748b56\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " Sep 05 20:00:33.037182 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037014 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tt6f\" (UniqueName: \"kubernetes.io/projected/592a6634-6158-4697-b423-0407a3748b56-kube-api-access-4tt6f\") pod \"592a6634-6158-4697-b423-0407a3748b56\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " Sep 05 20:00:33.037182 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037041 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-workload-socket\") pod \"592a6634-6158-4697-b423-0407a3748b56\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " Sep 05 20:00:33.037182 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037079 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-istio-envoy\") pod \"592a6634-6158-4697-b423-0407a3748b56\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " Sep 05 20:00:33.037182 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037107 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-credential-socket\") pod \"592a6634-6158-4697-b423-0407a3748b56\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " Sep 05 20:00:33.037182 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037153 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592a6634-6158-4697-b423-0407a3748b56-istio-podinfo\") pod \"592a6634-6158-4697-b423-0407a3748b56\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " Sep 05 20:00:33.037504 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037229 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592a6634-6158-4697-b423-0407a3748b56-istiod-ca-cert\") pod \"592a6634-6158-4697-b423-0407a3748b56\" (UID: \"592a6634-6158-4697-b423-0407a3748b56\") " Sep 05 20:00:33.037504 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037277 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "592a6634-6158-4697-b423-0407a3748b56" (UID: "592a6634-6158-4697-b423-0407a3748b56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:00:33.037504 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037291 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-istio-data" (OuterVolumeSpecName: "istio-data") pod "592a6634-6158-4697-b423-0407a3748b56" (UID: "592a6634-6158-4697-b423-0407a3748b56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:00:33.037504 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqtf\" (UniqueName: \"kubernetes.io/projected/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3-kube-api-access-bgqtf\") pod \"payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm\" (UID: \"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3\") " pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" Sep 05 20:00:33.037504 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037480 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "592a6634-6158-4697-b423-0407a3748b56" (UID: "592a6634-6158-4697-b423-0407a3748b56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:00:33.037754 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037628 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3-plugins-config-volume\") pod \"payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm\" (UID: \"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3\") " pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" Sep 05 20:00:33.037754 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037700 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:00:33.037754 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037717 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:00:33.037754 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037729 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:00:33.037988 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.037967 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "592a6634-6158-4697-b423-0407a3748b56" (UID: "592a6634-6158-4697-b423-0407a3748b56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:00:33.038167 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.038141 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592a6634-6158-4697-b423-0407a3748b56-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "592a6634-6158-4697-b423-0407a3748b56" (UID: "592a6634-6158-4697-b423-0407a3748b56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:00:33.039734 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.039668 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "592a6634-6158-4697-b423-0407a3748b56" (UID: "592a6634-6158-4697-b423-0407a3748b56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:00:33.039997 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.039966 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592a6634-6158-4697-b423-0407a3748b56-istio-token" (OuterVolumeSpecName: "istio-token") pod "592a6634-6158-4697-b423-0407a3748b56" (UID: "592a6634-6158-4697-b423-0407a3748b56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:00:33.039997 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.039973 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/592a6634-6158-4697-b423-0407a3748b56-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "592a6634-6158-4697-b423-0407a3748b56" (UID: "592a6634-6158-4697-b423-0407a3748b56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 05 20:00:33.040237 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.040216 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592a6634-6158-4697-b423-0407a3748b56-kube-api-access-4tt6f" (OuterVolumeSpecName: "kube-api-access-4tt6f") pod "592a6634-6158-4697-b423-0407a3748b56" (UID: "592a6634-6158-4697-b423-0407a3748b56"). InnerVolumeSpecName "kube-api-access-4tt6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:00:33.138667 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.138629 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqtf\" (UniqueName: \"kubernetes.io/projected/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3-kube-api-access-bgqtf\") pod \"payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm\" (UID: \"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3\") " pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" Sep 05 20:00:33.138879 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.138724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3-plugins-config-volume\") pod \"payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm\" (UID: \"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3\") " pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" Sep 05 20:00:33.138879 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.138760 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592a6634-6158-4697-b423-0407a3748b56-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:00:33.138879 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.138770 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592a6634-6158-4697-b423-0407a3748b56-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:00:33.138879 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.138779 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tt6f\" (UniqueName: \"kubernetes.io/projected/592a6634-6158-4697-b423-0407a3748b56-kube-api-access-4tt6f\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:00:33.138879 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.138788 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:00:33.138879 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.138824 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592a6634-6158-4697-b423-0407a3748b56-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:00:33.138879 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.138836 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592a6634-6158-4697-b423-0407a3748b56-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:00:33.139442 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.139420 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3-plugins-config-volume\") pod \"payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm\" (UID: \"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3\") " pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" Sep 05 20:00:33.156470 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.156438 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqtf\" (UniqueName: \"kubernetes.io/projected/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3-kube-api-access-bgqtf\") pod \"payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm\" (UID: \"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3\") " pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" Sep 05 20:00:33.200513 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.200474 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" Sep 05 20:00:33.477625 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.477415 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm"] Sep 05 20:00:33.482430 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:00:33.482394 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb960c2f7_b0d8_4a29_ae99_f7d2151a09e3.slice/crio-d08f1d623d030a457c89436af97cc4bcb8599c171a4aa9ae6eda0461250b33f1 WatchSource:0}: Error finding container d08f1d623d030a457c89436af97cc4bcb8599c171a4aa9ae6eda0461250b33f1: Status 404 returned error can't find the container with id d08f1d623d030a457c89436af97cc4bcb8599c171a4aa9ae6eda0461250b33f1 Sep 05 20:00:33.659078 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.659042 2578 generic.go:356] "Generic (PLEG): container finished" podID="592a6634-6158-4697-b423-0407a3748b56" containerID="77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563" exitCode=0 Sep 05 20:00:33.659277 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.659097 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" Sep 05 20:00:33.659277 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.659135 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" event={"ID":"592a6634-6158-4697-b423-0407a3748b56","Type":"ContainerDied","Data":"77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563"} Sep 05 20:00:33.659277 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.659215 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc" event={"ID":"592a6634-6158-4697-b423-0407a3748b56","Type":"ContainerDied","Data":"cf969f17e98a7c3c9e2c1b231357f621e9d3cf6aa764176da562c566553b5834"} Sep 05 20:00:33.659277 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.659252 2578 scope.go:117] "RemoveContainer" containerID="77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563" Sep 05 20:00:33.661002 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.660969 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" event={"ID":"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3","Type":"ContainerStarted","Data":"616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71"} Sep 05 20:00:33.661147 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.661060 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" Sep 05 20:00:33.661147 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.661086 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" event={"ID":"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3","Type":"ContainerStarted","Data":"d08f1d623d030a457c89436af97cc4bcb8599c171a4aa9ae6eda0461250b33f1"} Sep 05 20:00:33.669462 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.669443 2578 scope.go:117] "RemoveContainer" containerID="4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d" Sep 05 20:00:33.677360 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.677343 2578 scope.go:117] "RemoveContainer" containerID="77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563" Sep 05 20:00:33.677608 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:00:33.677589 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563\": container with ID starting with 77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563 not found: ID does not exist" containerID="77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563" Sep 05 20:00:33.677667 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.677616 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563"} err="failed to get container status \"77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563\": rpc error: code = NotFound desc = could not find container \"77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563\": container with ID starting with 77b498df77cb7ea71d2e7f7380508a9fa42437fa9f7737a4e72184847c2d7563 not found: ID does not exist" Sep 05 20:00:33.677667 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.677632 2578 scope.go:117] "RemoveContainer" containerID="4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d" Sep 05 20:00:33.677899 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:00:33.677885 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d\": container with ID starting with 4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d not found: ID does not exist" containerID="4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d" Sep 05 20:00:33.677948 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.677903 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d"} err="failed to get container status \"4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d\": rpc error: code = NotFound desc = could not find container \"4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d\": container with ID starting with 4fb2c321aa1ac44eaa4d9f637769438b8375d8a71851855e5e00badf953dc08d not found: ID does not exist" Sep 05 20:00:33.682735 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.682693 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" podStartSLOduration=1.682679229 podStartE2EDuration="1.682679229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:00:33.681336485 +0000 UTC m=+1054.166637368" watchObservedRunningTime="2026-09-05 20:00:33.682679229 +0000 UTC m=+1054.167980117" Sep 05 20:00:33.696915 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.696893 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc"] Sep 05 20:00:33.702689 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:33.702666 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e1fbf0a6-gw-openshift-default-78fc4f4cbd-pvtvc"] Sep 05 20:00:34.012658 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:34.012623 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="592a6634-6158-4697-b423-0407a3748b56" path="/var/lib/kubelet/pods/592a6634-6158-4697-b423-0407a3748b56/volumes" Sep 05 20:00:39.835867 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:39.835823 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:39.836263 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:39.835883 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:39.848862 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:39.848828 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:40.711468 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:40.711437 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:00:44.669153 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:00:44.669115 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" Sep 05 20:01:00.041793 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:00.041715 2578 scope.go:117] "RemoveContainer" containerID="e78a8c227c002ee664e3f2a314421f6658acac2c1d1d8f3e5acb7bb1e77bd532" Sep 05 20:01:01.129021 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.128989 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2"] Sep 05 20:01:01.129583 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.129562 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" Sep 05 20:01:01.129656 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.129587 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" Sep 05 20:01:01.129656 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.129628 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" Sep 05 20:01:01.129656 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.129637 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" Sep 05 20:01:01.129820 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.129749 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" Sep 05 20:01:01.129820 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.129763 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="592a6634-6158-4697-b423-0407a3748b56" containerName="istio-proxy" Sep 05 20:01:01.134379 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.134358 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.139454 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.139427 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-588081\"/\"default-dockercfg-jh7lg\"" Sep 05 20:01:01.139607 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.139526 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-588081\"/\"test-model-588081-kserve-self-signed-certs\"" Sep 05 20:01:01.139607 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.139586 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-588081\"/\"kube-root-ca.crt\"" Sep 05 20:01:01.140983 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.140961 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-588081\"/\"openshift-service-ca.crt\"" Sep 05 20:01:01.159839 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.159783 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2"] Sep 05 20:01:01.196211 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.196169 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4lt\" (UniqueName: \"kubernetes.io/projected/f5f18267-77a4-4f3b-9e44-799a83e0fa69-kube-api-access-4g4lt\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.196378 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.196220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-home\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.196378 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.196237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-dshm\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.196378 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.196332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-tmp-dir\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.196487 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.196429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-model-cache\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.196487 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.196468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f18267-77a4-4f3b-9e44-799a83e0fa69-tls-certs\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.297791 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.297750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f18267-77a4-4f3b-9e44-799a83e0fa69-tls-certs\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.297981 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.297813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4lt\" (UniqueName: \"kubernetes.io/projected/f5f18267-77a4-4f3b-9e44-799a83e0fa69-kube-api-access-4g4lt\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.297981 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.297860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-home\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.297981 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.297874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-dshm\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.297981 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.297911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-tmp-dir\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.297981 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.297960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-model-cache\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.298358 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.298320 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-home\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.298492 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.298384 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-model-cache\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.298492 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.298411 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-tmp-dir\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.300273 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.300254 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-dshm\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.300556 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.300537 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f18267-77a4-4f3b-9e44-799a83e0fa69-tls-certs\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.313675 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.313646 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4lt\" (UniqueName: \"kubernetes.io/projected/f5f18267-77a4-4f3b-9e44-799a83e0fa69-kube-api-access-4g4lt\") pod \"test-model-588081-kserve-d848d96fc-r6pg2\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.444566 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.444469 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:01.721434 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.721405 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2"] Sep 05 20:01:01.723439 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:01:01.723415 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5f18267_77a4_4f3b_9e44_799a83e0fa69.slice/crio-9b2d629e09b012bb4c368786f1580d8c3f08ad4f987752342577e2effe8ad267 WatchSource:0}: Error finding container 9b2d629e09b012bb4c368786f1580d8c3f08ad4f987752342577e2effe8ad267: Status 404 returned error can't find the container with id 9b2d629e09b012bb4c368786f1580d8c3f08ad4f987752342577e2effe8ad267 Sep 05 20:01:01.805062 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.805018 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" event={"ID":"f5f18267-77a4-4f3b-9e44-799a83e0fa69","Type":"ContainerStarted","Data":"97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879"} Sep 05 20:01:01.805258 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.805079 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" event={"ID":"f5f18267-77a4-4f3b-9e44-799a83e0fa69","Type":"ContainerStarted","Data":"9b2d629e09b012bb4c368786f1580d8c3f08ad4f987752342577e2effe8ad267"} Sep 05 20:01:01.827148 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:01.827097 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" podStartSLOduration=0.827079784 podStartE2EDuration="827.079784ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:01:01.82622184 +0000 UTC m=+1082.311522739" watchObservedRunningTime="2026-09-05 20:01:01.827079784 +0000 UTC m=+1082.312380657" Sep 05 20:01:02.748006 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:02.745639 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm"] Sep 05 20:01:02.748006 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:02.745985 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" podUID="b960c2f7-b0d8-4a29-ae99-f7d2151a09e3" containerName="payload-processing" containerID="cri-o://616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71" gracePeriod=30 Sep 05 20:01:03.022118 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.022084 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" Sep 05 20:01:03.115329 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.115293 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgqtf\" (UniqueName: \"kubernetes.io/projected/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3-kube-api-access-bgqtf\") pod \"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3\" (UID: \"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3\") " Sep 05 20:01:03.115329 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.115338 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3-plugins-config-volume\") pod \"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3\" (UID: \"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3\") " Sep 05 20:01:03.115781 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.115749 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b960c2f7-b0d8-4a29-ae99-f7d2151a09e3" (UID: "b960c2f7-b0d8-4a29-ae99-f7d2151a09e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:01:03.117552 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.117530 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3-kube-api-access-bgqtf" (OuterVolumeSpecName: "kube-api-access-bgqtf") pod "b960c2f7-b0d8-4a29-ae99-f7d2151a09e3" (UID: "b960c2f7-b0d8-4a29-ae99-f7d2151a09e3"). InnerVolumeSpecName "kube-api-access-bgqtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:01:03.216769 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.216733 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqtf\" (UniqueName: \"kubernetes.io/projected/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3-kube-api-access-bgqtf\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:03.216769 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.216761 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:03.817813 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.817774 2578 generic.go:356] "Generic (PLEG): container finished" podID="b960c2f7-b0d8-4a29-ae99-f7d2151a09e3" containerID="616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71" exitCode=0 Sep 05 20:01:03.818218 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.817843 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" Sep 05 20:01:03.818218 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.817840 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" event={"ID":"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3","Type":"ContainerDied","Data":"616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71"} Sep 05 20:01:03.818218 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.817906 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm" event={"ID":"b960c2f7-b0d8-4a29-ae99-f7d2151a09e3","Type":"ContainerDied","Data":"d08f1d623d030a457c89436af97cc4bcb8599c171a4aa9ae6eda0461250b33f1"} Sep 05 20:01:03.818218 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.817938 2578 scope.go:117] "RemoveContainer" containerID="616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71" Sep 05 20:01:03.826911 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.826893 2578 scope.go:117] "RemoveContainer" containerID="616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71" Sep 05 20:01:03.827229 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:01:03.827204 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71\": container with ID starting with 616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71 not found: ID does not exist" containerID="616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71" Sep 05 20:01:03.827272 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.827242 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71"} err="failed to get container status \"616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71\": rpc error: code = NotFound desc = could not find container \"616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71\": container with ID starting with 616983d9b87b0567013530cd04924ede951a78b0eab5b7068fb845a597d40d71 not found: ID does not exist" Sep 05 20:01:03.843559 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.843525 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm"] Sep 05 20:01:03.850875 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:03.850847 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a7428d13-5664958bb7-sxwsm"] Sep 05 20:01:04.013132 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:04.013097 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b960c2f7-b0d8-4a29-ae99-f7d2151a09e3" path="/var/lib/kubelet/pods/b960c2f7-b0d8-4a29-ae99-f7d2151a09e3/volumes" Sep 05 20:01:07.881973 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:07.881935 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk"] Sep 05 20:01:08.375774 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.375724 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p"] Sep 05 20:01:08.376404 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.376375 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b960c2f7-b0d8-4a29-ae99-f7d2151a09e3" containerName="payload-processing" Sep 05 20:01:08.376404 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.376403 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b960c2f7-b0d8-4a29-ae99-f7d2151a09e3" containerName="payload-processing" Sep 05 20:01:08.376585 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.376524 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b960c2f7-b0d8-4a29-ae99-f7d2151a09e3" containerName="payload-processing" Sep 05 20:01:08.381026 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.381002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:08.383525 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.383505 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c06406ce-serving-cert\"" Sep 05 20:01:08.392560 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.392533 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p"] Sep 05 20:01:08.466196 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.466154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-metrics-certs\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:08.466386 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.466222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-maas-api-tls\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:08.466386 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.466272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhsw7\" (UniqueName: \"kubernetes.io/projected/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-kube-api-access-bhsw7\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:08.466386 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.466337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-openshift-service-ca\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:08.567414 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.567370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-maas-api-tls\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:08.567619 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.567434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhsw7\" (UniqueName: \"kubernetes.io/projected/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-kube-api-access-bhsw7\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:08.567619 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.567478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-openshift-service-ca\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:08.567619 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:01:08.567515 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-serving-cert: secret "maas-api-e2e-derive-c06406ce-serving-cert" not found Sep 05 20:01:08.567619 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:01:08.567581 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-maas-api-tls podName:cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c nodeName:}" failed. No retries permitted until 2026-09-05 20:01:09.067564201 +0000 UTC m=+1089.552865058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-maas-api-tls") pod "maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" (UID: "cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c") : secret "maas-api-e2e-derive-c06406ce-serving-cert" not found Sep 05 20:01:08.567892 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.567663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-metrics-certs\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:08.568246 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.568224 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-openshift-service-ca\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:08.570075 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.570053 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-metrics-certs\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:08.577117 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:08.577098 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhsw7\" (UniqueName: \"kubernetes.io/projected/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-kube-api-access-bhsw7\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:09.074035 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:09.073994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-maas-api-tls\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:09.076458 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:09.076437 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-maas-api-tls\") pod \"maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:09.294883 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:09.294842 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:09.568231 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:09.568179 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p"] Sep 05 20:01:09.570646 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:01:09.570617 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7dd4f9_8bcb_4e17_921c_9bede8a0eb1c.slice/crio-ceeded8a24afe3385b3e00de88584defa336d429b9196bb7ea72a15c17a73237 WatchSource:0}: Error finding container ceeded8a24afe3385b3e00de88584defa336d429b9196bb7ea72a15c17a73237: Status 404 returned error can't find the container with id ceeded8a24afe3385b3e00de88584defa336d429b9196bb7ea72a15c17a73237 Sep 05 20:01:09.855286 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:09.855250 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" event={"ID":"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c","Type":"ContainerStarted","Data":"ceeded8a24afe3385b3e00de88584defa336d429b9196bb7ea72a15c17a73237"} Sep 05 20:01:10.861851 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:10.861816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" event={"ID":"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c","Type":"ContainerStarted","Data":"9e3044231751f4bc002f10a6a47d3ff042c96fe8a15589c5874e379ed54b557d"} Sep 05 20:01:10.862261 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:10.861919 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:10.880287 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:10.880230 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" podStartSLOduration=2.441655081 podStartE2EDuration="2.880212597s" totalImagesPullingTime="438.557516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:01:10.879959713 +0000 UTC m=+1091.365260586" watchObservedRunningTime="2026-09-05 20:01:10.880212597 +0000 UTC m=+1091.365513518" Sep 05 20:01:11.445199 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:11.445162 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:11.445362 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:11.445214 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:11.457932 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:11.457904 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:11.877924 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:11.877890 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:01:12.302741 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.302708 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2"] Sep 05 20:01:12.302957 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.302918 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" podUID="c21bad62-a2c3-47ea-9a01-3a5b4598e5c5" containerName="payload-pre-processing" containerID="cri-o://2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c" gracePeriod=30 Sep 05 20:01:12.571226 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.571157 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" Sep 05 20:01:12.708619 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.708579 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5-plugins-config-volume\") pod \"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5\" (UID: \"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5\") " Sep 05 20:01:12.708791 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.708663 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qltxj\" (UniqueName: \"kubernetes.io/projected/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5-kube-api-access-qltxj\") pod \"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5\" (UID: \"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5\") " Sep 05 20:01:12.709057 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.709031 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c21bad62-a2c3-47ea-9a01-3a5b4598e5c5" (UID: "c21bad62-a2c3-47ea-9a01-3a5b4598e5c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:01:12.710770 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.710748 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5-kube-api-access-qltxj" (OuterVolumeSpecName: "kube-api-access-qltxj") pod "c21bad62-a2c3-47ea-9a01-3a5b4598e5c5" (UID: "c21bad62-a2c3-47ea-9a01-3a5b4598e5c5"). InnerVolumeSpecName "kube-api-access-qltxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:01:12.809976 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.809933 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:12.809976 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.809966 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qltxj\" (UniqueName: \"kubernetes.io/projected/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5-kube-api-access-qltxj\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:12.873677 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.873589 2578 generic.go:356] "Generic (PLEG): container finished" podID="c21bad62-a2c3-47ea-9a01-3a5b4598e5c5" containerID="2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c" exitCode=0 Sep 05 20:01:12.873677 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.873638 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" Sep 05 20:01:12.873883 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.873679 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" event={"ID":"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5","Type":"ContainerDied","Data":"2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c"} Sep 05 20:01:12.873883 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.873761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2" event={"ID":"c21bad62-a2c3-47ea-9a01-3a5b4598e5c5","Type":"ContainerDied","Data":"710f2aab074b1aa2fe5be442594feefe058e59e7603e500a8221d7a6d1430f08"} Sep 05 20:01:12.873883 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.873792 2578 scope.go:117] "RemoveContainer" containerID="2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c" Sep 05 20:01:12.883583 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.883566 2578 scope.go:117] "RemoveContainer" containerID="2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c" Sep 05 20:01:12.883892 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:01:12.883872 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c\": container with ID starting with 2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c not found: ID does not exist" containerID="2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c" Sep 05 20:01:12.883944 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.883898 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c"} err="failed to get container status \"2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c\": rpc error: code = NotFound desc = could not find container \"2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c\": container with ID starting with 2108dd1da7412db29bec199acffcbe90e9e5c5a46d72f313778e27d6e0d3cb8c not found: ID does not exist" Sep 05 20:01:12.898448 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.898413 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2"] Sep 05 20:01:12.903465 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:12.903438 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1fd18b-7fcbf946db-jbzw2"] Sep 05 20:01:14.013008 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:14.012976 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c21bad62-a2c3-47ea-9a01-3a5b4598e5c5" path="/var/lib/kubelet/pods/c21bad62-a2c3-47ea-9a01-3a5b4598e5c5/volumes" Sep 05 20:01:16.872220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:16.872192 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:22.901628 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:22.901560 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" podUID="76c47bd2-9436-471b-b03e-77e1c32256da" containerName="main" containerID="cri-o://42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706" gracePeriod=45 Sep 05 20:01:23.159264 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.159197 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:01:23.311245 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.311214 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-dshm\") pod \"76c47bd2-9436-471b-b03e-77e1c32256da\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " Sep 05 20:01:23.311440 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.311257 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-home\") pod \"76c47bd2-9436-471b-b03e-77e1c32256da\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " Sep 05 20:01:23.311440 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.311299 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c47bd2-9436-471b-b03e-77e1c32256da-tls-certs\") pod \"76c47bd2-9436-471b-b03e-77e1c32256da\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " Sep 05 20:01:23.311440 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.311387 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-model-cache\") pod \"76c47bd2-9436-471b-b03e-77e1c32256da\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " Sep 05 20:01:23.311617 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.311453 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd7bl\" (UniqueName: \"kubernetes.io/projected/76c47bd2-9436-471b-b03e-77e1c32256da-kube-api-access-zd7bl\") pod \"76c47bd2-9436-471b-b03e-77e1c32256da\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " Sep 05 20:01:23.311617 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.311505 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-tmp-dir\") pod \"76c47bd2-9436-471b-b03e-77e1c32256da\" (UID: \"76c47bd2-9436-471b-b03e-77e1c32256da\") " Sep 05 20:01:23.311725 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.311585 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-home" (OuterVolumeSpecName: "home") pod "76c47bd2-9436-471b-b03e-77e1c32256da" (UID: "76c47bd2-9436-471b-b03e-77e1c32256da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:01:23.311725 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.311648 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-model-cache" (OuterVolumeSpecName: "model-cache") pod "76c47bd2-9436-471b-b03e-77e1c32256da" (UID: "76c47bd2-9436-471b-b03e-77e1c32256da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:01:23.311855 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.311835 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76c47bd2-9436-471b-b03e-77e1c32256da" (UID: "76c47bd2-9436-471b-b03e-77e1c32256da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:01:23.311911 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.311866 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:23.311911 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.311885 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:23.313720 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.313693 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-dshm" (OuterVolumeSpecName: "dshm") pod "76c47bd2-9436-471b-b03e-77e1c32256da" (UID: "76c47bd2-9436-471b-b03e-77e1c32256da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:01:23.313852 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.313735 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c47bd2-9436-471b-b03e-77e1c32256da-kube-api-access-zd7bl" (OuterVolumeSpecName: "kube-api-access-zd7bl") pod "76c47bd2-9436-471b-b03e-77e1c32256da" (UID: "76c47bd2-9436-471b-b03e-77e1c32256da"). InnerVolumeSpecName "kube-api-access-zd7bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:01:23.313852 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.313748 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c47bd2-9436-471b-b03e-77e1c32256da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76c47bd2-9436-471b-b03e-77e1c32256da" (UID: "76c47bd2-9436-471b-b03e-77e1c32256da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:01:23.413302 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.413204 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd7bl\" (UniqueName: \"kubernetes.io/projected/76c47bd2-9436-471b-b03e-77e1c32256da-kube-api-access-zd7bl\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:23.413302 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.413236 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:23.413302 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.413249 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c47bd2-9436-471b-b03e-77e1c32256da-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:23.413302 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.413257 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c47bd2-9436-471b-b03e-77e1c32256da-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:23.933005 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.932970 2578 generic.go:356] "Generic (PLEG): container finished" podID="76c47bd2-9436-471b-b03e-77e1c32256da" containerID="42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706" exitCode=0 Sep 05 20:01:23.933410 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.933011 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" Sep 05 20:01:23.933410 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.933058 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" event={"ID":"76c47bd2-9436-471b-b03e-77e1c32256da","Type":"ContainerDied","Data":"42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706"} Sep 05 20:01:23.933410 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.933096 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk" event={"ID":"76c47bd2-9436-471b-b03e-77e1c32256da","Type":"ContainerDied","Data":"2b065a4a13efdfc01810dc55d08733d08f7fd17e5adacd3194e34c1cb9ae0fff"} Sep 05 20:01:23.933410 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.933113 2578 scope.go:117] "RemoveContainer" containerID="42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706" Sep 05 20:01:23.942668 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.942648 2578 scope.go:117] "RemoveContainer" containerID="42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706" Sep 05 20:01:23.942963 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:01:23.942943 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706\": container with ID starting with 42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706 not found: ID does not exist" containerID="42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706" Sep 05 20:01:23.943031 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.942972 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706"} err="failed to get container status \"42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706\": rpc error: code = NotFound desc = could not find container \"42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706\": container with ID starting with 42bcf2ac9ea80189af032d37acc699cf54d412d01bacaa736ba0ff36dc389706 not found: ID does not exist" Sep 05 20:01:23.955432 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.955408 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk"] Sep 05 20:01:23.959541 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:23.959522 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-1fd18b/autores-model-1fd18b-kserve-75b7dfcc95-tvmbk"] Sep 05 20:01:24.012625 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:24.012592 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76c47bd2-9436-471b-b03e-77e1c32256da" path="/var/lib/kubelet/pods/76c47bd2-9436-471b-b03e-77e1c32256da/volumes" Sep 05 20:01:38.804660 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:38.804625 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p"] Sep 05 20:01:38.805268 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:38.804905 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" podUID="cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c" containerName="maas-api" containerID="cri-o://9e3044231751f4bc002f10a6a47d3ff042c96fe8a15589c5874e379ed54b557d" gracePeriod=30 Sep 05 20:01:39.010332 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.010298 2578 generic.go:356] "Generic (PLEG): container finished" podID="cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c" containerID="9e3044231751f4bc002f10a6a47d3ff042c96fe8a15589c5874e379ed54b557d" exitCode=0 Sep 05 20:01:39.010487 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.010354 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" event={"ID":"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c","Type":"ContainerDied","Data":"9e3044231751f4bc002f10a6a47d3ff042c96fe8a15589c5874e379ed54b557d"} Sep 05 20:01:39.057683 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.057619 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:39.168881 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.168849 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-maas-api-tls\") pod \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " Sep 05 20:01:39.169065 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.168903 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-openshift-service-ca\") pod \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " Sep 05 20:01:39.169065 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.168982 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-metrics-certs\") pod \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " Sep 05 20:01:39.169179 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.169095 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhsw7\" (UniqueName: \"kubernetes.io/projected/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-kube-api-access-bhsw7\") pod \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\" (UID: \"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c\") " Sep 05 20:01:39.169335 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.169303 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c" (UID: "cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:01:39.169486 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.169381 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-openshift-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:39.172222 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.172184 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c" (UID: "cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:01:39.172314 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.172210 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-kube-api-access-bhsw7" (OuterVolumeSpecName: "kube-api-access-bhsw7") pod "cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c" (UID: "cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c"). InnerVolumeSpecName "kube-api-access-bhsw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:01:39.172314 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.172217 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c" (UID: "cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:01:39.269932 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.269897 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:39.269932 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.269924 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhsw7\" (UniqueName: \"kubernetes.io/projected/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-kube-api-access-bhsw7\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:39.269932 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:39.269934 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c-maas-api-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:01:40.016162 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:40.016124 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" event={"ID":"cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c","Type":"ContainerDied","Data":"ceeded8a24afe3385b3e00de88584defa336d429b9196bb7ea72a15c17a73237"} Sep 05 20:01:40.016576 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:40.016196 2578 scope.go:117] "RemoveContainer" containerID="9e3044231751f4bc002f10a6a47d3ff042c96fe8a15589c5874e379ed54b557d" Sep 05 20:01:40.016576 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:40.016143 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p" Sep 05 20:01:40.044513 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:40.044485 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p"] Sep 05 20:01:40.049240 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:40.049213 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c06406ce-75f679bcd5-rk87p"] Sep 05 20:01:42.012101 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:01:42.012061 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c" path="/var/lib/kubelet/pods/cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c/volumes" Sep 05 20:02:00.917086 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.917052 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm"] Sep 05 20:02:00.917496 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.917412 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c47bd2-9436-471b-b03e-77e1c32256da" containerName="main" Sep 05 20:02:00.917496 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.917423 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c47bd2-9436-471b-b03e-77e1c32256da" containerName="main" Sep 05 20:02:00.917496 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.917437 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c21bad62-a2c3-47ea-9a01-3a5b4598e5c5" containerName="payload-pre-processing" Sep 05 20:02:00.917496 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.917443 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c21bad62-a2c3-47ea-9a01-3a5b4598e5c5" containerName="payload-pre-processing" Sep 05 20:02:00.917496 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.917458 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c" containerName="maas-api" Sep 05 20:02:00.917496 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.917465 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c" containerName="maas-api" Sep 05 20:02:00.917688 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.917520 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb7dd4f9-8bcb-4e17-921c-9bede8a0eb1c" containerName="maas-api" Sep 05 20:02:00.917688 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.917530 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c21bad62-a2c3-47ea-9a01-3a5b4598e5c5" containerName="payload-pre-processing" Sep 05 20:02:00.917688 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.917535 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c47bd2-9436-471b-b03e-77e1c32256da" containerName="main" Sep 05 20:02:00.922003 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.921982 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" Sep 05 20:02:00.924780 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.924755 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-49af4f\"" Sep 05 20:02:00.924907 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.924759 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-49af4f-dockercfg-f7x25\"" Sep 05 20:02:00.932607 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:00.932577 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm"] Sep 05 20:02:01.091347 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:01.091304 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5lms\" (UniqueName: \"kubernetes.io/projected/7e97e5c9-4946-4a7d-b662-e78e51c00d4e-kube-api-access-w5lms\") pod \"payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm\" (UID: \"7e97e5c9-4946-4a7d-b662-e78e51c00d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" Sep 05 20:02:01.091545 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:01.091363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e97e5c9-4946-4a7d-b662-e78e51c00d4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm\" (UID: \"7e97e5c9-4946-4a7d-b662-e78e51c00d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" Sep 05 20:02:01.192668 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:01.192570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e97e5c9-4946-4a7d-b662-e78e51c00d4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm\" (UID: \"7e97e5c9-4946-4a7d-b662-e78e51c00d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" Sep 05 20:02:01.192973 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:01.192946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5lms\" (UniqueName: \"kubernetes.io/projected/7e97e5c9-4946-4a7d-b662-e78e51c00d4e-kube-api-access-w5lms\") pod \"payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm\" (UID: \"7e97e5c9-4946-4a7d-b662-e78e51c00d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" Sep 05 20:02:01.193416 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:01.193395 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e97e5c9-4946-4a7d-b662-e78e51c00d4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm\" (UID: \"7e97e5c9-4946-4a7d-b662-e78e51c00d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" Sep 05 20:02:01.201677 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:01.201645 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5lms\" (UniqueName: \"kubernetes.io/projected/7e97e5c9-4946-4a7d-b662-e78e51c00d4e-kube-api-access-w5lms\") pod \"payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm\" (UID: \"7e97e5c9-4946-4a7d-b662-e78e51c00d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" Sep 05 20:02:01.233427 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:01.233388 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" Sep 05 20:02:01.507893 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:01.507862 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm"] Sep 05 20:02:01.510278 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:02:01.510242 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e97e5c9_4946_4a7d_b662_e78e51c00d4e.slice/crio-7fde9e41c0a2f1d5c39a3c1b1a760f0ca2eb88902d992da65bdc65f6821c7453 WatchSource:0}: Error finding container 7fde9e41c0a2f1d5c39a3c1b1a760f0ca2eb88902d992da65bdc65f6821c7453: Status 404 returned error can't find the container with id 7fde9e41c0a2f1d5c39a3c1b1a760f0ca2eb88902d992da65bdc65f6821c7453 Sep 05 20:02:02.140986 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:02.140928 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" event={"ID":"7e97e5c9-4946-4a7d-b662-e78e51c00d4e","Type":"ContainerStarted","Data":"59d255682b6c63252817332b913fd6f0cbd546e4e15dbdc3585d9f3346bbe593"} Sep 05 20:02:02.141455 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:02.141045 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" Sep 05 20:02:02.141455 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:02.141070 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" event={"ID":"7e97e5c9-4946-4a7d-b662-e78e51c00d4e","Type":"ContainerStarted","Data":"7fde9e41c0a2f1d5c39a3c1b1a760f0ca2eb88902d992da65bdc65f6821c7453"} Sep 05 20:02:02.164311 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:02.164251 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" podStartSLOduration=2.164233844 podStartE2EDuration="2.164233844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:02:02.162342267 +0000 UTC m=+1142.647643140" watchObservedRunningTime="2026-09-05 20:02:02.164233844 +0000 UTC m=+1142.649534718" Sep 05 20:02:13.148384 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:13.148345 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" Sep 05 20:02:52.012966 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:52.012934 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2"] Sep 05 20:02:53.137603 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.137565 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm"] Sep 05 20:02:53.138025 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.137777 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" podUID="7e97e5c9-4946-4a7d-b662-e78e51c00d4e" containerName="payload-pre-processing" containerID="cri-o://59d255682b6c63252817332b913fd6f0cbd546e4e15dbdc3585d9f3346bbe593" gracePeriod=30 Sep 05 20:02:53.147169 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.147131 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" podUID="7e97e5c9-4946-4a7d-b662-e78e51c00d4e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.51:9004: connect: connection refused" Sep 05 20:02:53.390541 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.390460 2578 generic.go:356] "Generic (PLEG): container finished" podID="7e97e5c9-4946-4a7d-b662-e78e51c00d4e" containerID="59d255682b6c63252817332b913fd6f0cbd546e4e15dbdc3585d9f3346bbe593" exitCode=0 Sep 05 20:02:53.390541 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.390517 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" event={"ID":"7e97e5c9-4946-4a7d-b662-e78e51c00d4e","Type":"ContainerDied","Data":"59d255682b6c63252817332b913fd6f0cbd546e4e15dbdc3585d9f3346bbe593"} Sep 05 20:02:53.390785 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.390563 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" event={"ID":"7e97e5c9-4946-4a7d-b662-e78e51c00d4e","Type":"ContainerDied","Data":"7fde9e41c0a2f1d5c39a3c1b1a760f0ca2eb88902d992da65bdc65f6821c7453"} Sep 05 20:02:53.390785 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.390580 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fde9e41c0a2f1d5c39a3c1b1a760f0ca2eb88902d992da65bdc65f6821c7453" Sep 05 20:02:53.394323 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.394300 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" Sep 05 20:02:53.497192 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.497150 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5lms\" (UniqueName: \"kubernetes.io/projected/7e97e5c9-4946-4a7d-b662-e78e51c00d4e-kube-api-access-w5lms\") pod \"7e97e5c9-4946-4a7d-b662-e78e51c00d4e\" (UID: \"7e97e5c9-4946-4a7d-b662-e78e51c00d4e\") " Sep 05 20:02:53.497401 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.497285 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e97e5c9-4946-4a7d-b662-e78e51c00d4e-plugins-config-volume\") pod \"7e97e5c9-4946-4a7d-b662-e78e51c00d4e\" (UID: \"7e97e5c9-4946-4a7d-b662-e78e51c00d4e\") " Sep 05 20:02:53.497651 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.497626 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e97e5c9-4946-4a7d-b662-e78e51c00d4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e97e5c9-4946-4a7d-b662-e78e51c00d4e" (UID: "7e97e5c9-4946-4a7d-b662-e78e51c00d4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:02:53.499329 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.499305 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e97e5c9-4946-4a7d-b662-e78e51c00d4e-kube-api-access-w5lms" (OuterVolumeSpecName: "kube-api-access-w5lms") pod "7e97e5c9-4946-4a7d-b662-e78e51c00d4e" (UID: "7e97e5c9-4946-4a7d-b662-e78e51c00d4e"). InnerVolumeSpecName "kube-api-access-w5lms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:02:53.598493 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.598454 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e97e5c9-4946-4a7d-b662-e78e51c00d4e-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:02:53.598493 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:53.598484 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5lms\" (UniqueName: \"kubernetes.io/projected/7e97e5c9-4946-4a7d-b662-e78e51c00d4e-kube-api-access-w5lms\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:02:54.396413 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:54.396330 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm" Sep 05 20:02:54.415714 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:54.415680 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm"] Sep 05 20:02:54.420259 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:54.420235 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-49af4f-76787666cd-fj4pm"] Sep 05 20:02:56.012377 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:02:56.012343 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e97e5c9-4946-4a7d-b662-e78e51c00d4e" path="/var/lib/kubelet/pods/7e97e5c9-4946-4a7d-b662-e78e51c00d4e/volumes" Sep 05 20:03:00.034631 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:00.034600 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 20:03:00.039064 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:00.039043 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 20:03:03.682831 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:03.682779 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t"] Sep 05 20:03:03.683300 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:03.683086 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" podUID="0a15528c-00ca-4405-8e46-72e5b206c3b6" containerName="maas-api" containerID="cri-o://26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214" gracePeriod=30 Sep 05 20:03:04.133013 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.132985 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 20:03:04.204230 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.204194 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-metrics-certs\") pod \"0a15528c-00ca-4405-8e46-72e5b206c3b6\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " Sep 05 20:03:04.204421 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.204300 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8p4x\" (UniqueName: \"kubernetes.io/projected/0a15528c-00ca-4405-8e46-72e5b206c3b6-kube-api-access-k8p4x\") pod \"0a15528c-00ca-4405-8e46-72e5b206c3b6\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " Sep 05 20:03:04.204421 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.204343 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a15528c-00ca-4405-8e46-72e5b206c3b6-openshift-service-ca\") pod \"0a15528c-00ca-4405-8e46-72e5b206c3b6\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " Sep 05 20:03:04.204421 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.204362 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-maas-api-tls\") pod \"0a15528c-00ca-4405-8e46-72e5b206c3b6\" (UID: \"0a15528c-00ca-4405-8e46-72e5b206c3b6\") " Sep 05 20:03:04.204753 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.204720 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a15528c-00ca-4405-8e46-72e5b206c3b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a15528c-00ca-4405-8e46-72e5b206c3b6" (UID: "0a15528c-00ca-4405-8e46-72e5b206c3b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:03:04.206443 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.206411 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a15528c-00ca-4405-8e46-72e5b206c3b6" (UID: "0a15528c-00ca-4405-8e46-72e5b206c3b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:03:04.206542 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.206451 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a15528c-00ca-4405-8e46-72e5b206c3b6-kube-api-access-k8p4x" (OuterVolumeSpecName: "kube-api-access-k8p4x") pod "0a15528c-00ca-4405-8e46-72e5b206c3b6" (UID: "0a15528c-00ca-4405-8e46-72e5b206c3b6"). InnerVolumeSpecName "kube-api-access-k8p4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:03:04.206542 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.206465 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0a15528c-00ca-4405-8e46-72e5b206c3b6" (UID: "0a15528c-00ca-4405-8e46-72e5b206c3b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:03:04.305183 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.305143 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8p4x\" (UniqueName: \"kubernetes.io/projected/0a15528c-00ca-4405-8e46-72e5b206c3b6-kube-api-access-k8p4x\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:04.305183 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.305176 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a15528c-00ca-4405-8e46-72e5b206c3b6-openshift-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:04.305183 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.305186 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-maas-api-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:04.305419 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.305194 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a15528c-00ca-4405-8e46-72e5b206c3b6-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:04.447035 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.446995 2578 generic.go:356] "Generic (PLEG): container finished" podID="0a15528c-00ca-4405-8e46-72e5b206c3b6" containerID="26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214" exitCode=0 Sep 05 20:03:04.447199 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.447045 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" Sep 05 20:03:04.447199 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.447084 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" event={"ID":"0a15528c-00ca-4405-8e46-72e5b206c3b6","Type":"ContainerDied","Data":"26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214"} Sep 05 20:03:04.447199 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.447136 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t" event={"ID":"0a15528c-00ca-4405-8e46-72e5b206c3b6","Type":"ContainerDied","Data":"e216e503758f81b82a6b95d4a60afd9207e890678a9d933ac1873c3d8a828887"} Sep 05 20:03:04.447199 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.447166 2578 scope.go:117] "RemoveContainer" containerID="26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214" Sep 05 20:03:04.456639 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.456622 2578 scope.go:117] "RemoveContainer" containerID="26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214" Sep 05 20:03:04.456950 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:03:04.456929 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214\": container with ID starting with 26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214 not found: ID does not exist" containerID="26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214" Sep 05 20:03:04.457006 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.456962 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214"} err="failed to get container status \"26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214\": rpc error: code = NotFound desc = could not find container \"26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214\": container with ID starting with 26e4b195f48edbc64a72b68c765bc88e080ccd41a19d711b67ea48e68abc8214 not found: ID does not exist" Sep 05 20:03:04.469649 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.469592 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t"] Sep 05 20:03:04.474307 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:04.474278 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-588081-578dfb4d48-x472t"] Sep 05 20:03:06.012814 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:06.012770 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a15528c-00ca-4405-8e46-72e5b206c3b6" path="/var/lib/kubelet/pods/0a15528c-00ca-4405-8e46-72e5b206c3b6/volumes" Sep 05 20:03:07.032920 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.032882 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" podUID="f5f18267-77a4-4f3b-9e44-799a83e0fa69" containerName="main" containerID="cri-o://97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879" gracePeriod=45 Sep 05 20:03:07.287482 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.287412 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:03:07.435784 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.435738 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-tmp-dir\") pod \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " Sep 05 20:03:07.435983 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.435835 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-dshm\") pod \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " Sep 05 20:03:07.435983 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.435870 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-home\") pod \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " Sep 05 20:03:07.435983 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.435915 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f18267-77a4-4f3b-9e44-799a83e0fa69-tls-certs\") pod \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " Sep 05 20:03:07.435983 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.435974 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g4lt\" (UniqueName: \"kubernetes.io/projected/f5f18267-77a4-4f3b-9e44-799a83e0fa69-kube-api-access-4g4lt\") pod \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " Sep 05 20:03:07.436158 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.436030 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-model-cache\") pod \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\" (UID: \"f5f18267-77a4-4f3b-9e44-799a83e0fa69\") " Sep 05 20:03:07.436158 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.436043 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5f18267-77a4-4f3b-9e44-799a83e0fa69" (UID: "f5f18267-77a4-4f3b-9e44-799a83e0fa69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:03:07.436247 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.436150 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-home" (OuterVolumeSpecName: "home") pod "f5f18267-77a4-4f3b-9e44-799a83e0fa69" (UID: "f5f18267-77a4-4f3b-9e44-799a83e0fa69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:03:07.436335 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.436318 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:07.436335 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.436321 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5f18267-77a4-4f3b-9e44-799a83e0fa69" (UID: "f5f18267-77a4-4f3b-9e44-799a83e0fa69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:03:07.436438 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.436337 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:07.438173 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.438147 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f18267-77a4-4f3b-9e44-799a83e0fa69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5f18267-77a4-4f3b-9e44-799a83e0fa69" (UID: "f5f18267-77a4-4f3b-9e44-799a83e0fa69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:03:07.438295 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.438226 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f18267-77a4-4f3b-9e44-799a83e0fa69-kube-api-access-4g4lt" (OuterVolumeSpecName: "kube-api-access-4g4lt") pod "f5f18267-77a4-4f3b-9e44-799a83e0fa69" (UID: "f5f18267-77a4-4f3b-9e44-799a83e0fa69"). InnerVolumeSpecName "kube-api-access-4g4lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:03:07.438295 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.438234 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-dshm" (OuterVolumeSpecName: "dshm") pod "f5f18267-77a4-4f3b-9e44-799a83e0fa69" (UID: "f5f18267-77a4-4f3b-9e44-799a83e0fa69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:03:07.469704 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.469663 2578 generic.go:356] "Generic (PLEG): container finished" podID="f5f18267-77a4-4f3b-9e44-799a83e0fa69" containerID="97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879" exitCode=0 Sep 05 20:03:07.469922 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.469726 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" Sep 05 20:03:07.469922 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.469745 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" event={"ID":"f5f18267-77a4-4f3b-9e44-799a83e0fa69","Type":"ContainerDied","Data":"97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879"} Sep 05 20:03:07.469922 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.469818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2" event={"ID":"f5f18267-77a4-4f3b-9e44-799a83e0fa69","Type":"ContainerDied","Data":"9b2d629e09b012bb4c368786f1580d8c3f08ad4f987752342577e2effe8ad267"} Sep 05 20:03:07.469922 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.469831 2578 scope.go:117] "RemoveContainer" containerID="97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879" Sep 05 20:03:07.478976 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.478957 2578 scope.go:117] "RemoveContainer" containerID="97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879" Sep 05 20:03:07.479273 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:03:07.479254 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879\": container with ID starting with 97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879 not found: ID does not exist" containerID="97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879" Sep 05 20:03:07.479337 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.479286 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879"} err="failed to get container status \"97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879\": rpc error: code = NotFound desc = could not find container \"97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879\": container with ID starting with 97380dba68d399549c05b6a5e8f51562875db3dec64df24aac6e0478a326a879 not found: ID does not exist" Sep 05 20:03:07.492270 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.492228 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2"] Sep 05 20:03:07.495719 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.495689 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-588081/test-model-588081-kserve-d848d96fc-r6pg2"] Sep 05 20:03:07.537670 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.537589 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4lt\" (UniqueName: \"kubernetes.io/projected/f5f18267-77a4-4f3b-9e44-799a83e0fa69-kube-api-access-4g4lt\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:07.537670 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.537620 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:07.537670 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.537630 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f18267-77a4-4f3b-9e44-799a83e0fa69-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:07.537670 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:07.537640 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f18267-77a4-4f3b-9e44-799a83e0fa69-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:08.012027 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:08.011988 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f18267-77a4-4f3b-9e44-799a83e0fa69" path="/var/lib/kubelet/pods/f5f18267-77a4-4f3b-9e44-799a83e0fa69/volumes" Sep 05 20:03:18.820518 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.820480 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9"] Sep 05 20:03:18.821014 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.820994 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e97e5c9-4946-4a7d-b662-e78e51c00d4e" containerName="payload-pre-processing" Sep 05 20:03:18.821098 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.821017 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e97e5c9-4946-4a7d-b662-e78e51c00d4e" containerName="payload-pre-processing" Sep 05 20:03:18.821098 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.821036 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f18267-77a4-4f3b-9e44-799a83e0fa69" containerName="main" Sep 05 20:03:18.821098 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.821043 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f18267-77a4-4f3b-9e44-799a83e0fa69" containerName="main" Sep 05 20:03:18.821098 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.821057 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a15528c-00ca-4405-8e46-72e5b206c3b6" containerName="maas-api" Sep 05 20:03:18.821098 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.821062 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a15528c-00ca-4405-8e46-72e5b206c3b6" containerName="maas-api" Sep 05 20:03:18.821243 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.821139 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a15528c-00ca-4405-8e46-72e5b206c3b6" containerName="maas-api" Sep 05 20:03:18.821243 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.821146 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f18267-77a4-4f3b-9e44-799a83e0fa69" containerName="main" Sep 05 20:03:18.821243 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.821155 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e97e5c9-4946-4a7d-b662-e78e51c00d4e" containerName="payload-pre-processing" Sep 05 20:03:18.825699 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.825678 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:18.829353 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.829328 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fedc2772-serving-cert\"" Sep 05 20:03:18.845110 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.845060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjdh\" (UniqueName: \"kubernetes.io/projected/03f9bd34-d131-4c80-8db5-da53d78ca54b-kube-api-access-6xjdh\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:18.845267 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.845141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f9bd34-d131-4c80-8db5-da53d78ca54b-openshift-service-ca\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:18.845267 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.845188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-metrics-certs\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:18.845267 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.845237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-maas-api-tls\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:18.849443 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.849415 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9"] Sep 05 20:03:18.945824 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.945759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjdh\" (UniqueName: \"kubernetes.io/projected/03f9bd34-d131-4c80-8db5-da53d78ca54b-kube-api-access-6xjdh\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:18.946039 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.945836 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f9bd34-d131-4c80-8db5-da53d78ca54b-openshift-service-ca\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:18.946039 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.945869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-metrics-certs\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:18.946039 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.945892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-maas-api-tls\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:18.946039 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:03:18.946003 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-serving-cert: secret "maas-api-e2e-mt-fedc2772-serving-cert" not found Sep 05 20:03:18.946261 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:03:18.946056 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-maas-api-tls podName:03f9bd34-d131-4c80-8db5-da53d78ca54b nodeName:}" failed. No retries permitted until 2026-09-05 20:03:19.446041039 +0000 UTC m=+1219.931341890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-maas-api-tls") pod "maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" (UID: "03f9bd34-d131-4c80-8db5-da53d78ca54b") : secret "maas-api-e2e-mt-fedc2772-serving-cert" not found Sep 05 20:03:18.946629 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.946598 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f9bd34-d131-4c80-8db5-da53d78ca54b-openshift-service-ca\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:18.948494 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.948468 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-metrics-certs\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:18.964316 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:18.964282 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjdh\" (UniqueName: \"kubernetes.io/projected/03f9bd34-d131-4c80-8db5-da53d78ca54b-kube-api-access-6xjdh\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:19.449686 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:19.449650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-maas-api-tls\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:19.449895 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:03:19.449815 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-serving-cert: secret "maas-api-e2e-mt-fedc2772-serving-cert" not found Sep 05 20:03:19.449895 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:03:19.449883 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-maas-api-tls podName:03f9bd34-d131-4c80-8db5-da53d78ca54b nodeName:}" failed. No retries permitted until 2026-09-05 20:03:20.449867481 +0000 UTC m=+1220.935168332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-maas-api-tls") pod "maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" (UID: "03f9bd34-d131-4c80-8db5-da53d78ca54b") : secret "maas-api-e2e-mt-fedc2772-serving-cert" not found Sep 05 20:03:20.459578 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:20.459526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-maas-api-tls\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:20.462092 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:20.462065 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-maas-api-tls\") pod \"maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:20.636421 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:20.636374 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:20.911112 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:20.911078 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9"] Sep 05 20:03:20.913622 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:03:20.913596 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f9bd34_d131_4c80_8db5_da53d78ca54b.slice/crio-bd22f665ab2115ab8703b687619873091c56eafab44387ca08e01753518fdd64 WatchSource:0}: Error finding container bd22f665ab2115ab8703b687619873091c56eafab44387ca08e01753518fdd64: Status 404 returned error can't find the container with id bd22f665ab2115ab8703b687619873091c56eafab44387ca08e01753518fdd64 Sep 05 20:03:20.915090 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:20.915064 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 20:03:21.383171 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.383137 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk"] Sep 05 20:03:21.383554 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.383421 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" podUID="03c1f839-7fd4-4c3f-a65a-915f0689b1ce" containerName="maas-api" containerID="cri-o://bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc" gracePeriod=30 Sep 05 20:03:21.539995 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.539958 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" event={"ID":"03f9bd34-d131-4c80-8db5-da53d78ca54b","Type":"ContainerStarted","Data":"5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789"} Sep 05 20:03:21.540539 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.540025 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" event={"ID":"03f9bd34-d131-4c80-8db5-da53d78ca54b","Type":"ContainerStarted","Data":"bd22f665ab2115ab8703b687619873091c56eafab44387ca08e01753518fdd64"} Sep 05 20:03:21.540539 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.540061 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:21.558413 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.558296 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" podStartSLOduration=3.165120257 podStartE2EDuration="3.558278157s" totalImagesPullingTime="393.1579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:03:21.558091447 +0000 UTC m=+1222.043392315" watchObservedRunningTime="2026-09-05 20:03:21.558278157 +0000 UTC m=+1222.043579031" Sep 05 20:03:21.831721 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.831691 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 20:03:21.873448 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.873414 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-metrics-certs\") pod \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " Sep 05 20:03:21.873611 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.873482 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nt44\" (UniqueName: \"kubernetes.io/projected/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-kube-api-access-6nt44\") pod \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " Sep 05 20:03:21.873611 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.873527 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-maas-api-tls\") pod \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " Sep 05 20:03:21.873729 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.873662 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-openshift-service-ca\") pod \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\" (UID: \"03c1f839-7fd4-4c3f-a65a-915f0689b1ce\") " Sep 05 20:03:21.874035 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.874009 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03c1f839-7fd4-4c3f-a65a-915f0689b1ce" (UID: "03c1f839-7fd4-4c3f-a65a-915f0689b1ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:03:21.875645 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.875621 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03c1f839-7fd4-4c3f-a65a-915f0689b1ce" (UID: "03c1f839-7fd4-4c3f-a65a-915f0689b1ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:03:21.876071 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.876048 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-kube-api-access-6nt44" (OuterVolumeSpecName: "kube-api-access-6nt44") pod "03c1f839-7fd4-4c3f-a65a-915f0689b1ce" (UID: "03c1f839-7fd4-4c3f-a65a-915f0689b1ce"). InnerVolumeSpecName "kube-api-access-6nt44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:03:21.876141 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.876051 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "03c1f839-7fd4-4c3f-a65a-915f0689b1ce" (UID: "03c1f839-7fd4-4c3f-a65a-915f0689b1ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:03:21.975092 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.975053 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nt44\" (UniqueName: \"kubernetes.io/projected/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-kube-api-access-6nt44\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:21.975092 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.975088 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-maas-api-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:21.975350 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.975103 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-openshift-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:21.975350 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:21.975120 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c1f839-7fd4-4c3f-a65a-915f0689b1ce-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:03:22.545627 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:22.545595 2578 generic.go:356] "Generic (PLEG): container finished" podID="03c1f839-7fd4-4c3f-a65a-915f0689b1ce" containerID="bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc" exitCode=0 Sep 05 20:03:22.546087 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:22.545643 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" Sep 05 20:03:22.546087 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:22.545682 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" event={"ID":"03c1f839-7fd4-4c3f-a65a-915f0689b1ce","Type":"ContainerDied","Data":"bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc"} Sep 05 20:03:22.546087 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:22.545737 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk" event={"ID":"03c1f839-7fd4-4c3f-a65a-915f0689b1ce","Type":"ContainerDied","Data":"58775ce40140ad9cf7c057c281d85ebc136d2b072d78c592a5e7124511fb90e6"} Sep 05 20:03:22.546087 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:22.545764 2578 scope.go:117] "RemoveContainer" containerID="bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc" Sep 05 20:03:22.554476 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:22.554459 2578 scope.go:117] "RemoveContainer" containerID="bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc" Sep 05 20:03:22.554779 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:03:22.554750 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc\": container with ID starting with bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc not found: ID does not exist" containerID="bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc" Sep 05 20:03:22.554895 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:22.554786 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc"} err="failed to get container status \"bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc\": rpc error: code = NotFound desc = could not find container \"bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc\": container with ID starting with bd43cdde7d1b07dcd7b258ba51a3b60ba72f85e2e51f827da5c97efddd8f2adc not found: ID does not exist" Sep 05 20:03:22.565106 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:22.565076 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk"] Sep 05 20:03:22.568661 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:22.568629 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1dc7b1-5cc6f877df-mwdsk"] Sep 05 20:03:24.012718 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:24.012680 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03c1f839-7fd4-4c3f-a65a-915f0689b1ce" path="/var/lib/kubelet/pods/03c1f839-7fd4-4c3f-a65a-915f0689b1ce/volumes" Sep 05 20:03:27.552184 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:27.552154 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:03:33.109150 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.109113 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx"] Sep 05 20:03:33.109550 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.109528 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c1f839-7fd4-4c3f-a65a-915f0689b1ce" containerName="maas-api" Sep 05 20:03:33.109550 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.109544 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c1f839-7fd4-4c3f-a65a-915f0689b1ce" containerName="maas-api" Sep 05 20:03:33.109639 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.109601 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03c1f839-7fd4-4c3f-a65a-915f0689b1ce" containerName="maas-api" Sep 05 20:03:33.112651 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.112627 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.118519 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.118494 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-4d1885-serving-cert\"" Sep 05 20:03:33.133685 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.133651 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx"] Sep 05 20:03:33.173365 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.173326 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.173566 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.173376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.173566 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.173526 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.173690 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.173597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb6jk\" (UniqueName: \"kubernetes.io/projected/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-kube-api-access-jb6jk\") pod \"maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.274547 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.274506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.274734 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.274601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.274734 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.274633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb6jk\" (UniqueName: \"kubernetes.io/projected/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-kube-api-access-jb6jk\") pod \"maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.274734 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.274657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.275350 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.275318 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.277150 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.277124 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.277240 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.277159 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.284722 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.284696 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb6jk\" (UniqueName: \"kubernetes.io/projected/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-kube-api-access-jb6jk\") pod \"maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.423483 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.423388 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:33.701158 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:33.701054 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx"] Sep 05 20:03:33.706389 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:03:33.706354 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1d2e7d_0ae8_45c2_b290_b1b32bf4d3ab.slice/crio-4d02e22598c5ad0afd7c0ec3975db169fdd7c2ae3f8a657e89e033eed83d9931 WatchSource:0}: Error finding container 4d02e22598c5ad0afd7c0ec3975db169fdd7c2ae3f8a657e89e033eed83d9931: Status 404 returned error can't find the container with id 4d02e22598c5ad0afd7c0ec3975db169fdd7c2ae3f8a657e89e033eed83d9931 Sep 05 20:03:34.608352 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:34.608314 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" event={"ID":"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab","Type":"ContainerStarted","Data":"4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63"} Sep 05 20:03:34.608740 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:34.608371 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" event={"ID":"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab","Type":"ContainerStarted","Data":"4d02e22598c5ad0afd7c0ec3975db169fdd7c2ae3f8a657e89e033eed83d9931"} Sep 05 20:03:34.608740 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:34.608425 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:03:34.627267 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:34.627207 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" podStartSLOduration=1.151518636 podStartE2EDuration="1.627189307s" totalImagesPullingTime="475.670671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:03:34.625992658 +0000 UTC m=+1235.111293530" watchObservedRunningTime="2026-09-05 20:03:34.627189307 +0000 UTC m=+1235.112490180" Sep 05 20:03:40.619114 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:03:40.619084 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:04:00.418303 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.418215 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9"] Sep 05 20:04:00.418813 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.418469 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" podUID="03f9bd34-d131-4c80-8db5-da53d78ca54b" containerName="maas-api" containerID="cri-o://5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789" gracePeriod=30 Sep 05 20:04:00.667530 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.667501 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:04:00.745452 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.745361 2578 generic.go:356] "Generic (PLEG): container finished" podID="03f9bd34-d131-4c80-8db5-da53d78ca54b" containerID="5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789" exitCode=0 Sep 05 20:04:00.745452 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.745408 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" Sep 05 20:04:00.745627 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.745463 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" event={"ID":"03f9bd34-d131-4c80-8db5-da53d78ca54b","Type":"ContainerDied","Data":"5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789"} Sep 05 20:04:00.745627 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.745513 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9" event={"ID":"03f9bd34-d131-4c80-8db5-da53d78ca54b","Type":"ContainerDied","Data":"bd22f665ab2115ab8703b687619873091c56eafab44387ca08e01753518fdd64"} Sep 05 20:04:00.745627 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.745540 2578 scope.go:117] "RemoveContainer" containerID="5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789" Sep 05 20:04:00.754659 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.754640 2578 scope.go:117] "RemoveContainer" containerID="5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789" Sep 05 20:04:00.755041 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:04:00.755017 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789\": container with ID starting with 5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789 not found: ID does not exist" containerID="5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789" Sep 05 20:04:00.755119 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.755052 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789"} err="failed to get container status \"5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789\": rpc error: code = NotFound desc = could not find container \"5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789\": container with ID starting with 5396b04c3362037266a0be1f1c5c62e17bd2de796b45a51cabed98f453ce2789 not found: ID does not exist" Sep 05 20:04:00.842822 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.842767 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xjdh\" (UniqueName: \"kubernetes.io/projected/03f9bd34-d131-4c80-8db5-da53d78ca54b-kube-api-access-6xjdh\") pod \"03f9bd34-d131-4c80-8db5-da53d78ca54b\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " Sep 05 20:04:00.842822 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.842828 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f9bd34-d131-4c80-8db5-da53d78ca54b-openshift-service-ca\") pod \"03f9bd34-d131-4c80-8db5-da53d78ca54b\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " Sep 05 20:04:00.843090 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.842864 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-metrics-certs\") pod \"03f9bd34-d131-4c80-8db5-da53d78ca54b\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " Sep 05 20:04:00.843090 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.842965 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-maas-api-tls\") pod \"03f9bd34-d131-4c80-8db5-da53d78ca54b\" (UID: \"03f9bd34-d131-4c80-8db5-da53d78ca54b\") " Sep 05 20:04:00.843269 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.843246 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f9bd34-d131-4c80-8db5-da53d78ca54b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03f9bd34-d131-4c80-8db5-da53d78ca54b" (UID: "03f9bd34-d131-4c80-8db5-da53d78ca54b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:04:00.844941 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.844911 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03f9bd34-d131-4c80-8db5-da53d78ca54b" (UID: "03f9bd34-d131-4c80-8db5-da53d78ca54b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:04:00.845416 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.845389 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "03f9bd34-d131-4c80-8db5-da53d78ca54b" (UID: "03f9bd34-d131-4c80-8db5-da53d78ca54b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:04:00.845416 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.845401 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f9bd34-d131-4c80-8db5-da53d78ca54b-kube-api-access-6xjdh" (OuterVolumeSpecName: "kube-api-access-6xjdh") pod "03f9bd34-d131-4c80-8db5-da53d78ca54b" (UID: "03f9bd34-d131-4c80-8db5-da53d78ca54b"). InnerVolumeSpecName "kube-api-access-6xjdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:04:00.943984 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.943940 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjdh\" (UniqueName: \"kubernetes.io/projected/03f9bd34-d131-4c80-8db5-da53d78ca54b-kube-api-access-6xjdh\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:04:00.943984 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.943977 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f9bd34-d131-4c80-8db5-da53d78ca54b-openshift-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:04:00.943984 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.943993 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:04:00.944227 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:00.944005 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9bd34-d131-4c80-8db5-da53d78ca54b-maas-api-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:04:01.069429 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.069391 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9"] Sep 05 20:04:01.073728 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.073700 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fedc2772-745bcbb7c6-r62f9"] Sep 05 20:04:01.799840 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.799808 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv"] Sep 05 20:04:01.800258 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.800242 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f9bd34-d131-4c80-8db5-da53d78ca54b" containerName="maas-api" Sep 05 20:04:01.800318 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.800263 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f9bd34-d131-4c80-8db5-da53d78ca54b" containerName="maas-api" Sep 05 20:04:01.800356 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.800346 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f9bd34-d131-4c80-8db5-da53d78ca54b" containerName="maas-api" Sep 05 20:04:01.805243 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.805217 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:01.808120 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.808078 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-0cc31a-serving-cert\"" Sep 05 20:04:01.817150 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.817121 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv"] Sep 05 20:04:01.952655 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.952614 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:01.952655 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.952661 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbkb\" (UniqueName: \"kubernetes.io/projected/dfd4067e-ad23-44b8-9f41-c27b73760a47-kube-api-access-nlbkb\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:01.952920 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.952689 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4067e-ad23-44b8-9f41-c27b73760a47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:01.952920 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:01.952745 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:02.013163 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:02.013127 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f9bd34-d131-4c80-8db5-da53d78ca54b" path="/var/lib/kubelet/pods/03f9bd34-d131-4c80-8db5-da53d78ca54b/volumes" Sep 05 20:04:02.054155 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:02.054055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4067e-ad23-44b8-9f41-c27b73760a47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:02.054155 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:02.054095 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:02.054386 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:02.054175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:02.054386 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:02.054214 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbkb\" (UniqueName: \"kubernetes.io/projected/dfd4067e-ad23-44b8-9f41-c27b73760a47-kube-api-access-nlbkb\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:02.054386 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:04:02.054347 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-serving-cert: secret "maas-api-e2e-shared-b-w3-0cc31a-serving-cert" not found Sep 05 20:04:02.054541 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:04:02.054427 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-maas-api-tls podName:dfd4067e-ad23-44b8-9f41-c27b73760a47 nodeName:}" failed. No retries permitted until 2026-09-05 20:04:02.554405823 +0000 UTC m=+1263.039706692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-maas-api-tls") pod "maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" (UID: "dfd4067e-ad23-44b8-9f41-c27b73760a47") : secret "maas-api-e2e-shared-b-w3-0cc31a-serving-cert" not found Sep 05 20:04:02.054791 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:02.054767 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4067e-ad23-44b8-9f41-c27b73760a47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:02.056549 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:02.056524 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:02.067157 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:02.067133 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbkb\" (UniqueName: \"kubernetes.io/projected/dfd4067e-ad23-44b8-9f41-c27b73760a47-kube-api-access-nlbkb\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:02.558576 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:02.558534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:02.561731 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:02.561697 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:02.719335 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:02.719294 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:02.999579 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:02.999552 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv"] Sep 05 20:04:03.001460 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:04:03.001433 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd4067e_ad23_44b8_9f41_c27b73760a47.slice/crio-a6aca551366ef424fbdcc8bfc4f1942ed4da738fd9d5f93667384d8cf3599d6d WatchSource:0}: Error finding container a6aca551366ef424fbdcc8bfc4f1942ed4da738fd9d5f93667384d8cf3599d6d: Status 404 returned error can't find the container with id a6aca551366ef424fbdcc8bfc4f1942ed4da738fd9d5f93667384d8cf3599d6d Sep 05 20:04:03.762103 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:03.762064 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" event={"ID":"dfd4067e-ad23-44b8-9f41-c27b73760a47","Type":"ContainerStarted","Data":"17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757"} Sep 05 20:04:03.762308 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:03.762143 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" event={"ID":"dfd4067e-ad23-44b8-9f41-c27b73760a47","Type":"ContainerStarted","Data":"a6aca551366ef424fbdcc8bfc4f1942ed4da738fd9d5f93667384d8cf3599d6d"} Sep 05 20:04:03.762308 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:03.762178 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:03.782497 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:03.782445 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" podStartSLOduration=2.424874217 podStartE2EDuration="2.782428756s" totalImagesPullingTime="357.554539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:04:03.780524684 +0000 UTC m=+1264.265825557" watchObservedRunningTime="2026-09-05 20:04:03.782428756 +0000 UTC m=+1264.267729679" Sep 05 20:04:09.772634 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:09.772605 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:04:25.927871 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:25.927837 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8"] Sep 05 20:04:25.931626 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:25.931607 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" Sep 05 20:04:25.934702 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:25.934673 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-1436b9\"" Sep 05 20:04:25.935031 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:25.935004 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-1436b9-dockercfg-qpn28\"" Sep 05 20:04:25.943058 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:25.943033 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8"] Sep 05 20:04:26.080485 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:26.080449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8\" (UID: \"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" Sep 05 20:04:26.080658 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:26.080614 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9n9\" (UniqueName: \"kubernetes.io/projected/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a-kube-api-access-dg9n9\") pod \"payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8\" (UID: \"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" Sep 05 20:04:26.181913 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:26.181821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9n9\" (UniqueName: \"kubernetes.io/projected/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a-kube-api-access-dg9n9\") pod \"payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8\" (UID: \"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" Sep 05 20:04:26.181913 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:26.181905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8\" (UID: \"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" Sep 05 20:04:26.182705 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:26.182681 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8\" (UID: \"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" Sep 05 20:04:26.190390 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:26.190362 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9n9\" (UniqueName: \"kubernetes.io/projected/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a-kube-api-access-dg9n9\") pod \"payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8\" (UID: \"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" Sep 05 20:04:26.242691 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:26.242655 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" Sep 05 20:04:26.522893 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:26.522862 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8"] Sep 05 20:04:26.524940 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:04:26.524912 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8d709b_3a6a_4a69_95eb_c44fe8125c3a.slice/crio-847556c24a98b37d34e119ad4a8edf30c493f1847f6be0d6d372d85e9267e6db WatchSource:0}: Error finding container 847556c24a98b37d34e119ad4a8edf30c493f1847f6be0d6d372d85e9267e6db: Status 404 returned error can't find the container with id 847556c24a98b37d34e119ad4a8edf30c493f1847f6be0d6d372d85e9267e6db Sep 05 20:04:26.876658 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:26.876553 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" event={"ID":"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a","Type":"ContainerStarted","Data":"c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2"} Sep 05 20:04:26.876658 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:26.876620 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" event={"ID":"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a","Type":"ContainerStarted","Data":"847556c24a98b37d34e119ad4a8edf30c493f1847f6be0d6d372d85e9267e6db"} Sep 05 20:04:26.876908 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:26.876694 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" Sep 05 20:04:26.895045 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:26.894980 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" podStartSLOduration=1.894963557 podStartE2EDuration="1.894963557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:04:26.893951967 +0000 UTC m=+1287.379252851" watchObservedRunningTime="2026-09-05 20:04:26.894963557 +0000 UTC m=+1287.380264430" Sep 05 20:04:37.883967 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:37.883934 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" Sep 05 20:04:53.780485 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:53.780435 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl"] Sep 05 20:04:53.784241 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:53.784221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" Sep 05 20:04:53.787593 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:53.787572 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ca4fba\"" Sep 05 20:04:53.787715 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:53.787677 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ca4fba-dockercfg-pfjf6\"" Sep 05 20:04:53.797535 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:53.797507 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl"] Sep 05 20:04:53.837157 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:53.837115 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0db3130-afc5-4873-89d8-9379c777d9e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl\" (UID: \"b0db3130-afc5-4873-89d8-9379c777d9e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" Sep 05 20:04:53.837329 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:53.837167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q56s8\" (UniqueName: \"kubernetes.io/projected/b0db3130-afc5-4873-89d8-9379c777d9e0-kube-api-access-q56s8\") pod \"payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl\" (UID: \"b0db3130-afc5-4873-89d8-9379c777d9e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" Sep 05 20:04:53.938553 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:53.938508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0db3130-afc5-4873-89d8-9379c777d9e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl\" (UID: \"b0db3130-afc5-4873-89d8-9379c777d9e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" Sep 05 20:04:53.938726 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:53.938561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q56s8\" (UniqueName: \"kubernetes.io/projected/b0db3130-afc5-4873-89d8-9379c777d9e0-kube-api-access-q56s8\") pod \"payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl\" (UID: \"b0db3130-afc5-4873-89d8-9379c777d9e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" Sep 05 20:04:53.939351 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:53.939330 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0db3130-afc5-4873-89d8-9379c777d9e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl\" (UID: \"b0db3130-afc5-4873-89d8-9379c777d9e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" Sep 05 20:04:53.947459 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:53.947434 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q56s8\" (UniqueName: \"kubernetes.io/projected/b0db3130-afc5-4873-89d8-9379c777d9e0-kube-api-access-q56s8\") pod \"payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl\" (UID: \"b0db3130-afc5-4873-89d8-9379c777d9e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" Sep 05 20:04:54.095416 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:54.095333 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" Sep 05 20:04:54.371673 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:54.371588 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl"] Sep 05 20:04:54.375565 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:04:54.375538 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0db3130_afc5_4873_89d8_9379c777d9e0.slice/crio-1aeec4ff1fc69f3ea43735b465dc5d39edb58bf994ee91daa799bc2ea86a4f22 WatchSource:0}: Error finding container 1aeec4ff1fc69f3ea43735b465dc5d39edb58bf994ee91daa799bc2ea86a4f22: Status 404 returned error can't find the container with id 1aeec4ff1fc69f3ea43735b465dc5d39edb58bf994ee91daa799bc2ea86a4f22 Sep 05 20:04:55.023087 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:55.023047 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" event={"ID":"b0db3130-afc5-4873-89d8-9379c777d9e0","Type":"ContainerStarted","Data":"57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a"} Sep 05 20:04:55.023563 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:55.023110 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" event={"ID":"b0db3130-afc5-4873-89d8-9379c777d9e0","Type":"ContainerStarted","Data":"1aeec4ff1fc69f3ea43735b465dc5d39edb58bf994ee91daa799bc2ea86a4f22"} Sep 05 20:04:55.023563 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:55.023141 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" Sep 05 20:04:55.041553 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:04:55.041501 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" podStartSLOduration=2.041482055 podStartE2EDuration="2.041482055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:04:55.040681082 +0000 UTC m=+1315.525981980" watchObservedRunningTime="2026-09-05 20:04:55.041482055 +0000 UTC m=+1315.526782929" Sep 05 20:05:06.030015 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:06.029970 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" Sep 05 20:05:17.557607 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:17.557568 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7842-140512613308000"] Sep 05 20:05:17.561430 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:17.561409 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:17.564629 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:17.564434 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cqkks\"" Sep 05 20:05:17.567913 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:17.567885 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-140512613308000"] Sep 05 20:05:17.665111 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:17.665071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkbrj\" (UniqueName: \"kubernetes.io/projected/50d62581-f1f5-463f-8670-226be06a69e4-kube-api-access-fkbrj\") pod \"test-curl-7842-140512613308000\" (UID: \"50d62581-f1f5-463f-8670-226be06a69e4\") " pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:17.766446 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:17.766406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkbrj\" (UniqueName: \"kubernetes.io/projected/50d62581-f1f5-463f-8670-226be06a69e4-kube-api-access-fkbrj\") pod \"test-curl-7842-140512613308000\" (UID: \"50d62581-f1f5-463f-8670-226be06a69e4\") " pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:17.777436 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:17.777404 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkbrj\" (UniqueName: \"kubernetes.io/projected/50d62581-f1f5-463f-8670-226be06a69e4-kube-api-access-fkbrj\") pod \"test-curl-7842-140512613308000\" (UID: \"50d62581-f1f5-463f-8670-226be06a69e4\") " pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:17.872729 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:17.872632 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:18.144080 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:18.144052 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-140512613308000"] Sep 05 20:05:18.147211 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:05:18.147179 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50d62581_f1f5_463f_8670_226be06a69e4.slice/crio-0ee167d168190f9920d85161616200b1a5196c4fcc59306e6821613a698c979a WatchSource:0}: Error finding container 0ee167d168190f9920d85161616200b1a5196c4fcc59306e6821613a698c979a: Status 404 returned error can't find the container with id 0ee167d168190f9920d85161616200b1a5196c4fcc59306e6821613a698c979a Sep 05 20:05:19.154037 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:19.153993 2578 generic.go:356] "Generic (PLEG): container finished" podID="50d62581-f1f5-463f-8670-226be06a69e4" containerID="cca9ca2b082e3d6cd80d09839b708407a5a1b1b92bdcf14f3036ab77abc0cc5d" exitCode=0 Sep 05 20:05:19.154528 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:19.154081 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-140512613308000" event={"ID":"50d62581-f1f5-463f-8670-226be06a69e4","Type":"ContainerDied","Data":"cca9ca2b082e3d6cd80d09839b708407a5a1b1b92bdcf14f3036ab77abc0cc5d"} Sep 05 20:05:19.154528 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:19.154120 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-140512613308000" event={"ID":"50d62581-f1f5-463f-8670-226be06a69e4","Type":"ContainerStarted","Data":"0ee167d168190f9920d85161616200b1a5196c4fcc59306e6821613a698c979a"} Sep 05 20:05:20.293364 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.293336 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:20.329568 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.329532 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7842-140512613308000_50d62581-f1f5-463f-8670-226be06a69e4/test-curl-7842-140512613308000/0.log" Sep 05 20:05:20.351066 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.351026 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7842-140512613308000"] Sep 05 20:05:20.355306 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.355280 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7842-140512613308000"] Sep 05 20:05:20.392702 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.392670 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkbrj\" (UniqueName: \"kubernetes.io/projected/50d62581-f1f5-463f-8670-226be06a69e4-kube-api-access-fkbrj\") pod \"50d62581-f1f5-463f-8670-226be06a69e4\" (UID: \"50d62581-f1f5-463f-8670-226be06a69e4\") " Sep 05 20:05:20.394734 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.394711 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d62581-f1f5-463f-8670-226be06a69e4-kube-api-access-fkbrj" (OuterVolumeSpecName: "kube-api-access-fkbrj") pod "50d62581-f1f5-463f-8670-226be06a69e4" (UID: "50d62581-f1f5-463f-8670-226be06a69e4"). InnerVolumeSpecName "kube-api-access-fkbrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:05:20.465528 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.465436 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7842-140512613308000"] Sep 05 20:05:20.465936 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.465913 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d62581-f1f5-463f-8670-226be06a69e4" containerName="test-curl-7842-140512613308000" Sep 05 20:05:20.466042 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.465939 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d62581-f1f5-463f-8670-226be06a69e4" containerName="test-curl-7842-140512613308000" Sep 05 20:05:20.466094 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.466064 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d62581-f1f5-463f-8670-226be06a69e4" containerName="test-curl-7842-140512613308000" Sep 05 20:05:20.469291 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.469272 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:20.473302 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.473279 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-140512613308000"] Sep 05 20:05:20.494166 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.494139 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkbrj\" (UniqueName: \"kubernetes.io/projected/50d62581-f1f5-463f-8670-226be06a69e4-kube-api-access-fkbrj\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:05:20.595550 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.595505 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjlz4\" (UniqueName: \"kubernetes.io/projected/d55c441e-59b3-4b5b-a540-7526c66c2b51-kube-api-access-hjlz4\") pod \"test-curl-7842-140512613308000\" (UID: \"d55c441e-59b3-4b5b-a540-7526c66c2b51\") " pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:20.696624 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.696589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjlz4\" (UniqueName: \"kubernetes.io/projected/d55c441e-59b3-4b5b-a540-7526c66c2b51-kube-api-access-hjlz4\") pod \"test-curl-7842-140512613308000\" (UID: \"d55c441e-59b3-4b5b-a540-7526c66c2b51\") " pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:20.705854 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.705828 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjlz4\" (UniqueName: \"kubernetes.io/projected/d55c441e-59b3-4b5b-a540-7526c66c2b51-kube-api-access-hjlz4\") pod \"test-curl-7842-140512613308000\" (UID: \"d55c441e-59b3-4b5b-a540-7526c66c2b51\") " pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:20.779892 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:20.779859 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:21.052407 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:21.052304 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-140512613308000"] Sep 05 20:05:21.059436 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:05:21.059408 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd55c441e_59b3_4b5b_a540_7526c66c2b51.slice/crio-b2e4e1c03ff34fde3ae82b2b5e27bc52b1fa53e1048cd50d0c0bb8fd904439cc WatchSource:0}: Error finding container b2e4e1c03ff34fde3ae82b2b5e27bc52b1fa53e1048cd50d0c0bb8fd904439cc: Status 404 returned error can't find the container with id b2e4e1c03ff34fde3ae82b2b5e27bc52b1fa53e1048cd50d0c0bb8fd904439cc Sep 05 20:05:21.164872 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:21.164831 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-140512613308000" event={"ID":"d55c441e-59b3-4b5b-a540-7526c66c2b51","Type":"ContainerStarted","Data":"b2e4e1c03ff34fde3ae82b2b5e27bc52b1fa53e1048cd50d0c0bb8fd904439cc"} Sep 05 20:05:21.166115 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:21.166097 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee167d168190f9920d85161616200b1a5196c4fcc59306e6821613a698c979a" Sep 05 20:05:21.166209 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:21.166114 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:21.178737 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:21.178701 2578 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7842-140512613308000" oldPodUID="50d62581-f1f5-463f-8670-226be06a69e4" podUID="d55c441e-59b3-4b5b-a540-7526c66c2b51" Sep 05 20:05:22.012720 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:22.012682 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d62581-f1f5-463f-8670-226be06a69e4" path="/var/lib/kubelet/pods/50d62581-f1f5-463f-8670-226be06a69e4/volumes" Sep 05 20:05:22.173544 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:22.173463 2578 generic.go:356] "Generic (PLEG): container finished" podID="d55c441e-59b3-4b5b-a540-7526c66c2b51" containerID="8cc241540cd32c1a1b2ec839de2f97859325ce0b87f6dd000ebde4b599e8df34" exitCode=0 Sep 05 20:05:22.173719 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:22.173551 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-140512613308000" event={"ID":"d55c441e-59b3-4b5b-a540-7526c66c2b51","Type":"ContainerDied","Data":"8cc241540cd32c1a1b2ec839de2f97859325ce0b87f6dd000ebde4b599e8df34"} Sep 05 20:05:23.308353 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.308320 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:23.329730 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.329697 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7842-140512613308000_d55c441e-59b3-4b5b-a540-7526c66c2b51/test-curl-7842-140512613308000/0.log" Sep 05 20:05:23.353049 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.353009 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7842-140512613308000"] Sep 05 20:05:23.355378 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.355353 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7842-140512613308000"] Sep 05 20:05:23.421348 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.421304 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjlz4\" (UniqueName: \"kubernetes.io/projected/d55c441e-59b3-4b5b-a540-7526c66c2b51-kube-api-access-hjlz4\") pod \"d55c441e-59b3-4b5b-a540-7526c66c2b51\" (UID: \"d55c441e-59b3-4b5b-a540-7526c66c2b51\") " Sep 05 20:05:23.423369 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.423346 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55c441e-59b3-4b5b-a540-7526c66c2b51-kube-api-access-hjlz4" (OuterVolumeSpecName: "kube-api-access-hjlz4") pod "d55c441e-59b3-4b5b-a540-7526c66c2b51" (UID: "d55c441e-59b3-4b5b-a540-7526c66c2b51"). InnerVolumeSpecName "kube-api-access-hjlz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:05:23.480040 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.480006 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7842-140512613308576"] Sep 05 20:05:23.480392 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.480379 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d55c441e-59b3-4b5b-a540-7526c66c2b51" containerName="test-curl-7842-140512613308000" Sep 05 20:05:23.480463 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.480393 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d55c441e-59b3-4b5b-a540-7526c66c2b51" containerName="test-curl-7842-140512613308000" Sep 05 20:05:23.480537 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.480463 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d55c441e-59b3-4b5b-a540-7526c66c2b51" containerName="test-curl-7842-140512613308000" Sep 05 20:05:23.483364 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.483349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:23.488904 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.488880 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-140512613308576"] Sep 05 20:05:23.522182 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.522150 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjlz4\" (UniqueName: \"kubernetes.io/projected/d55c441e-59b3-4b5b-a540-7526c66c2b51-kube-api-access-hjlz4\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:05:23.622975 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.622935 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmlm\" (UniqueName: \"kubernetes.io/projected/3b0a9aaa-9938-4677-bdd3-be2c7adea490-kube-api-access-jvmlm\") pod \"test-curl-7842-140512613308576\" (UID: \"3b0a9aaa-9938-4677-bdd3-be2c7adea490\") " pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:23.723501 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.723415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmlm\" (UniqueName: \"kubernetes.io/projected/3b0a9aaa-9938-4677-bdd3-be2c7adea490-kube-api-access-jvmlm\") pod \"test-curl-7842-140512613308576\" (UID: \"3b0a9aaa-9938-4677-bdd3-be2c7adea490\") " pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:23.732720 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.732690 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmlm\" (UniqueName: \"kubernetes.io/projected/3b0a9aaa-9938-4677-bdd3-be2c7adea490-kube-api-access-jvmlm\") pod \"test-curl-7842-140512613308576\" (UID: \"3b0a9aaa-9938-4677-bdd3-be2c7adea490\") " pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:23.792875 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:23.792836 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:24.012638 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:24.012605 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d55c441e-59b3-4b5b-a540-7526c66c2b51" path="/var/lib/kubelet/pods/d55c441e-59b3-4b5b-a540-7526c66c2b51/volumes" Sep 05 20:05:24.062780 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:24.062751 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-140512613308576"] Sep 05 20:05:24.064400 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:05:24.064369 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b0a9aaa_9938_4677_bdd3_be2c7adea490.slice/crio-a304572be7ab0c323f31269b45ca758d86fa397c9278b7400ebba2835ff0e8ac WatchSource:0}: Error finding container a304572be7ab0c323f31269b45ca758d86fa397c9278b7400ebba2835ff0e8ac: Status 404 returned error can't find the container with id a304572be7ab0c323f31269b45ca758d86fa397c9278b7400ebba2835ff0e8ac Sep 05 20:05:24.184159 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:24.184128 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308000" Sep 05 20:05:24.184316 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:24.184181 2578 scope.go:117] "RemoveContainer" containerID="8cc241540cd32c1a1b2ec839de2f97859325ce0b87f6dd000ebde4b599e8df34" Sep 05 20:05:24.185537 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:24.185469 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-140512613308576" event={"ID":"3b0a9aaa-9938-4677-bdd3-be2c7adea490","Type":"ContainerStarted","Data":"a304572be7ab0c323f31269b45ca758d86fa397c9278b7400ebba2835ff0e8ac"} Sep 05 20:05:25.192524 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:25.192493 2578 generic.go:356] "Generic (PLEG): container finished" podID="3b0a9aaa-9938-4677-bdd3-be2c7adea490" containerID="5b9d75f99679e85f86281229c5bb0b44ac814c34c9025dcd8018ba21013f4f7a" exitCode=0 Sep 05 20:05:25.192933 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:25.192529 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-140512613308576" event={"ID":"3b0a9aaa-9938-4677-bdd3-be2c7adea490","Type":"ContainerDied","Data":"5b9d75f99679e85f86281229c5bb0b44ac814c34c9025dcd8018ba21013f4f7a"} Sep 05 20:05:26.333174 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.333145 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:26.362074 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.362039 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7842-140512613308576_3b0a9aaa-9938-4677-bdd3-be2c7adea490/test-curl-7842-140512613308576/0.log" Sep 05 20:05:26.384153 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.384118 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7842-140512613308576"] Sep 05 20:05:26.388110 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.388079 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7842-140512613308576"] Sep 05 20:05:26.451575 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.451472 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvmlm\" (UniqueName: \"kubernetes.io/projected/3b0a9aaa-9938-4677-bdd3-be2c7adea490-kube-api-access-jvmlm\") pod \"3b0a9aaa-9938-4677-bdd3-be2c7adea490\" (UID: \"3b0a9aaa-9938-4677-bdd3-be2c7adea490\") " Sep 05 20:05:26.453611 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.453572 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0a9aaa-9938-4677-bdd3-be2c7adea490-kube-api-access-jvmlm" (OuterVolumeSpecName: "kube-api-access-jvmlm") pod "3b0a9aaa-9938-4677-bdd3-be2c7adea490" (UID: "3b0a9aaa-9938-4677-bdd3-be2c7adea490"). InnerVolumeSpecName "kube-api-access-jvmlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:05:26.505574 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.505534 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7842-140512613308576"] Sep 05 20:05:26.505987 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.505973 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0a9aaa-9938-4677-bdd3-be2c7adea490" containerName="test-curl-7842-140512613308576" Sep 05 20:05:26.506037 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.505989 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0a9aaa-9938-4677-bdd3-be2c7adea490" containerName="test-curl-7842-140512613308576" Sep 05 20:05:26.506078 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.506045 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b0a9aaa-9938-4677-bdd3-be2c7adea490" containerName="test-curl-7842-140512613308576" Sep 05 20:05:26.509008 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.508990 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:26.514599 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.514575 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-140512613308576"] Sep 05 20:05:26.552870 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.552833 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvmlm\" (UniqueName: \"kubernetes.io/projected/3b0a9aaa-9938-4677-bdd3-be2c7adea490-kube-api-access-jvmlm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:05:26.653922 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.653883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn2lm\" (UniqueName: \"kubernetes.io/projected/5a8468e9-bb03-44e8-9436-9e3b8e256992-kube-api-access-nn2lm\") pod \"test-curl-7842-140512613308576\" (UID: \"5a8468e9-bb03-44e8-9436-9e3b8e256992\") " pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:26.755054 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.755023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2lm\" (UniqueName: \"kubernetes.io/projected/5a8468e9-bb03-44e8-9436-9e3b8e256992-kube-api-access-nn2lm\") pod \"test-curl-7842-140512613308576\" (UID: \"5a8468e9-bb03-44e8-9436-9e3b8e256992\") " pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:26.764218 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.764192 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn2lm\" (UniqueName: \"kubernetes.io/projected/5a8468e9-bb03-44e8-9436-9e3b8e256992-kube-api-access-nn2lm\") pod \"test-curl-7842-140512613308576\" (UID: \"5a8468e9-bb03-44e8-9436-9e3b8e256992\") " pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:26.820671 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:26.820635 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:27.093621 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:27.093593 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-140512613308576"] Sep 05 20:05:27.095983 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:05:27.095948 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a8468e9_bb03_44e8_9436_9e3b8e256992.slice/crio-2027f90683a18f60d05c3a6d3bc1481672b0757dce87c3ee96ba81c0ca2eaf84 WatchSource:0}: Error finding container 2027f90683a18f60d05c3a6d3bc1481672b0757dce87c3ee96ba81c0ca2eaf84: Status 404 returned error can't find the container with id 2027f90683a18f60d05c3a6d3bc1481672b0757dce87c3ee96ba81c0ca2eaf84 Sep 05 20:05:27.205080 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:27.205046 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a304572be7ab0c323f31269b45ca758d86fa397c9278b7400ebba2835ff0e8ac" Sep 05 20:05:27.205080 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:27.205055 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:27.206387 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:27.206357 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-140512613308576" event={"ID":"5a8468e9-bb03-44e8-9436-9e3b8e256992","Type":"ContainerStarted","Data":"2027f90683a18f60d05c3a6d3bc1481672b0757dce87c3ee96ba81c0ca2eaf84"} Sep 05 20:05:27.218031 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:27.218003 2578 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7842-140512613308576" oldPodUID="3b0a9aaa-9938-4677-bdd3-be2c7adea490" podUID="5a8468e9-bb03-44e8-9436-9e3b8e256992" Sep 05 20:05:28.012978 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:28.012941 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b0a9aaa-9938-4677-bdd3-be2c7adea490" path="/var/lib/kubelet/pods/3b0a9aaa-9938-4677-bdd3-be2c7adea490/volumes" Sep 05 20:05:28.213078 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:28.212979 2578 generic.go:356] "Generic (PLEG): container finished" podID="5a8468e9-bb03-44e8-9436-9e3b8e256992" containerID="49c433b53ba6721c8bf0f6a6eb5302958bfd0ac421fbd26bfed7a923a92e911a" exitCode=0 Sep 05 20:05:28.213078 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:28.213022 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-140512613308576" event={"ID":"5a8468e9-bb03-44e8-9436-9e3b8e256992","Type":"ContainerDied","Data":"49c433b53ba6721c8bf0f6a6eb5302958bfd0ac421fbd26bfed7a923a92e911a"} Sep 05 20:05:29.347764 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:29.347740 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:29.370986 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:29.370957 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7842-140512613308576_5a8468e9-bb03-44e8-9436-9e3b8e256992/test-curl-7842-140512613308576/0.log" Sep 05 20:05:29.391836 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:29.391789 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7842-140512613308576"] Sep 05 20:05:29.395973 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:29.395940 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7842-140512613308576"] Sep 05 20:05:29.481400 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:29.481318 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn2lm\" (UniqueName: \"kubernetes.io/projected/5a8468e9-bb03-44e8-9436-9e3b8e256992-kube-api-access-nn2lm\") pod \"5a8468e9-bb03-44e8-9436-9e3b8e256992\" (UID: \"5a8468e9-bb03-44e8-9436-9e3b8e256992\") " Sep 05 20:05:29.483409 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:29.483381 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8468e9-bb03-44e8-9436-9e3b8e256992-kube-api-access-nn2lm" (OuterVolumeSpecName: "kube-api-access-nn2lm") pod "5a8468e9-bb03-44e8-9436-9e3b8e256992" (UID: "5a8468e9-bb03-44e8-9436-9e3b8e256992"). InnerVolumeSpecName "kube-api-access-nn2lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:05:29.582253 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:29.582213 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn2lm\" (UniqueName: \"kubernetes.io/projected/5a8468e9-bb03-44e8-9436-9e3b8e256992-kube-api-access-nn2lm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:05:30.013274 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:30.013236 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a8468e9-bb03-44e8-9436-9e3b8e256992" path="/var/lib/kubelet/pods/5a8468e9-bb03-44e8-9436-9e3b8e256992/volumes" Sep 05 20:05:30.223230 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:30.223200 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-140512613308576" Sep 05 20:05:30.223409 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:30.223253 2578 scope.go:117] "RemoveContainer" containerID="49c433b53ba6721c8bf0f6a6eb5302958bfd0ac421fbd26bfed7a923a92e911a" Sep 05 20:05:34.360078 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.360047 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm"] Sep 05 20:05:34.360537 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.360519 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8468e9-bb03-44e8-9436-9e3b8e256992" containerName="test-curl-7842-140512613308576" Sep 05 20:05:34.360599 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.360540 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8468e9-bb03-44e8-9436-9e3b8e256992" containerName="test-curl-7842-140512613308576" Sep 05 20:05:34.360666 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.360653 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8468e9-bb03-44e8-9436-9e3b8e256992" containerName="test-curl-7842-140512613308576" Sep 05 20:05:34.365182 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.365160 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:34.367927 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.367902 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ada83762-serving-cert\"" Sep 05 20:05:34.372351 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.372322 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm"] Sep 05 20:05:34.528143 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.528098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e6892a-633d-49c7-9a90-1d23b5c89a3a-openshift-service-ca\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:34.528332 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.528202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gdcj\" (UniqueName: \"kubernetes.io/projected/41e6892a-633d-49c7-9a90-1d23b5c89a3a-kube-api-access-2gdcj\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:34.528332 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.528264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-metrics-certs\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:34.528332 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.528329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-maas-api-tls\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:34.629640 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.629547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gdcj\" (UniqueName: \"kubernetes.io/projected/41e6892a-633d-49c7-9a90-1d23b5c89a3a-kube-api-access-2gdcj\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:34.629640 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.629593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-metrics-certs\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:34.629869 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.629656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-maas-api-tls\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:34.629869 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.629676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e6892a-633d-49c7-9a90-1d23b5c89a3a-openshift-service-ca\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:34.629869 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:05:34.629781 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-serving-cert: secret "maas-api-e2e-mt-ada83762-serving-cert" not found Sep 05 20:05:34.630042 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:05:34.629874 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-maas-api-tls podName:41e6892a-633d-49c7-9a90-1d23b5c89a3a nodeName:}" failed. No retries permitted until 2026-09-05 20:05:35.129856941 +0000 UTC m=+1355.615157796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-maas-api-tls") pod "maas-api-e2e-mt-ada83762-dd67dc884-g24gm" (UID: "41e6892a-633d-49c7-9a90-1d23b5c89a3a") : secret "maas-api-e2e-mt-ada83762-serving-cert" not found Sep 05 20:05:34.630440 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.630422 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e6892a-633d-49c7-9a90-1d23b5c89a3a-openshift-service-ca\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:34.632166 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.632150 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-metrics-certs\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:34.640621 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:34.640598 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gdcj\" (UniqueName: \"kubernetes.io/projected/41e6892a-633d-49c7-9a90-1d23b5c89a3a-kube-api-access-2gdcj\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:35.025643 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.025604 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2"] Sep 05 20:05:35.029440 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.029420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" Sep 05 20:05:35.032350 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.032306 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ada83762\"" Sep 05 20:05:35.032350 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.032332 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ada83762-dockercfg-btzrp\"" Sep 05 20:05:35.032621 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.032591 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4bk4\" (UniqueName: \"kubernetes.io/projected/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83-kube-api-access-x4bk4\") pod \"payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2\" (UID: \"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83\") " pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" Sep 05 20:05:35.032719 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.032700 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83-plugins-config-volume\") pod \"payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2\" (UID: \"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83\") " pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" Sep 05 20:05:35.041680 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.041651 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2"] Sep 05 20:05:35.133765 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.133721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4bk4\" (UniqueName: \"kubernetes.io/projected/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83-kube-api-access-x4bk4\") pod \"payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2\" (UID: \"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83\") " pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" Sep 05 20:05:35.133996 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.133778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-maas-api-tls\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:35.133996 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.133837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83-plugins-config-volume\") pod \"payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2\" (UID: \"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83\") " pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" Sep 05 20:05:35.134572 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.134549 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83-plugins-config-volume\") pod \"payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2\" (UID: \"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83\") " pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" Sep 05 20:05:35.136448 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.136418 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-maas-api-tls\") pod \"maas-api-e2e-mt-ada83762-dd67dc884-g24gm\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:35.142467 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.142424 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4bk4\" (UniqueName: \"kubernetes.io/projected/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83-kube-api-access-x4bk4\") pod \"payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2\" (UID: \"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83\") " pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" Sep 05 20:05:35.277964 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.277869 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:35.340344 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.340314 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" Sep 05 20:05:35.558562 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.558531 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm"] Sep 05 20:05:35.560653 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:05:35.560623 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e6892a_633d_49c7_9a90_1d23b5c89a3a.slice/crio-bae243315074d16ba8f29f7e099ba79d6b3951e75d8f6318acccea75d7aff387 WatchSource:0}: Error finding container bae243315074d16ba8f29f7e099ba79d6b3951e75d8f6318acccea75d7aff387: Status 404 returned error can't find the container with id bae243315074d16ba8f29f7e099ba79d6b3951e75d8f6318acccea75d7aff387 Sep 05 20:05:35.620750 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:35.620722 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2"] Sep 05 20:05:35.623332 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:05:35.623300 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb20b1e_ee6d_4c72_a079_b6f95edc1e83.slice/crio-1dd53badf58f478931a57832b3708b46a30d5c7959c8cad7e5fe4123574f9fe4 WatchSource:0}: Error finding container 1dd53badf58f478931a57832b3708b46a30d5c7959c8cad7e5fe4123574f9fe4: Status 404 returned error can't find the container with id 1dd53badf58f478931a57832b3708b46a30d5c7959c8cad7e5fe4123574f9fe4 Sep 05 20:05:36.257821 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:36.257769 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" event={"ID":"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83","Type":"ContainerStarted","Data":"3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936"} Sep 05 20:05:36.258009 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:36.257859 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" event={"ID":"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83","Type":"ContainerStarted","Data":"1dd53badf58f478931a57832b3708b46a30d5c7959c8cad7e5fe4123574f9fe4"} Sep 05 20:05:36.258009 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:36.257915 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" Sep 05 20:05:36.259379 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:36.259354 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" event={"ID":"41e6892a-633d-49c7-9a90-1d23b5c89a3a","Type":"ContainerStarted","Data":"48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1"} Sep 05 20:05:36.259482 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:36.259400 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" event={"ID":"41e6892a-633d-49c7-9a90-1d23b5c89a3a","Type":"ContainerStarted","Data":"bae243315074d16ba8f29f7e099ba79d6b3951e75d8f6318acccea75d7aff387"} Sep 05 20:05:36.259482 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:36.259431 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:36.278103 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:36.278047 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" podStartSLOduration=2.278027645 podStartE2EDuration="2.278027645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:05:36.276599891 +0000 UTC m=+1356.761900764" watchObservedRunningTime="2026-09-05 20:05:36.278027645 +0000 UTC m=+1356.763328519" Sep 05 20:05:36.294733 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:36.294684 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" podStartSLOduration=1.893441915 podStartE2EDuration="2.294663766s" totalImagesPullingTime="401.221851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:05:36.2940279 +0000 UTC m=+1356.779328776" watchObservedRunningTime="2026-09-05 20:05:36.294663766 +0000 UTC m=+1356.779964642" Sep 05 20:05:42.271130 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:42.271096 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:05:47.267350 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:05:47.267319 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" Sep 05 20:06:57.904609 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:57.904519 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm"] Sep 05 20:06:57.905139 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:57.904815 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" podUID="41e6892a-633d-49c7-9a90-1d23b5c89a3a" containerName="maas-api" containerID="cri-o://48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1" gracePeriod=30 Sep 05 20:06:57.960706 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:57.960669 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2"] Sep 05 20:06:57.960990 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:57.960938 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" podUID="2eb20b1e-ee6d-4c72-a079-b6f95edc1e83" containerName="payload-processing" containerID="cri-o://3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936" gracePeriod=30 Sep 05 20:06:58.222444 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.222417 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:06:58.255194 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.255168 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" Sep 05 20:06:58.337550 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.337515 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e6892a-633d-49c7-9a90-1d23b5c89a3a-openshift-service-ca\") pod \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " Sep 05 20:06:58.337550 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.337555 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4bk4\" (UniqueName: \"kubernetes.io/projected/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83-kube-api-access-x4bk4\") pod \"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83\" (UID: \"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83\") " Sep 05 20:06:58.337817 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.337594 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-metrics-certs\") pod \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " Sep 05 20:06:58.337817 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.337616 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83-plugins-config-volume\") pod \"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83\" (UID: \"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83\") " Sep 05 20:06:58.337817 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.337659 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gdcj\" (UniqueName: \"kubernetes.io/projected/41e6892a-633d-49c7-9a90-1d23b5c89a3a-kube-api-access-2gdcj\") pod \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " Sep 05 20:06:58.337817 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.337703 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-maas-api-tls\") pod \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\" (UID: \"41e6892a-633d-49c7-9a90-1d23b5c89a3a\") " Sep 05 20:06:58.338166 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.338138 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2eb20b1e-ee6d-4c72-a079-b6f95edc1e83" (UID: "2eb20b1e-ee6d-4c72-a079-b6f95edc1e83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:06:58.338257 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.338188 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e6892a-633d-49c7-9a90-1d23b5c89a3a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "41e6892a-633d-49c7-9a90-1d23b5c89a3a" (UID: "41e6892a-633d-49c7-9a90-1d23b5c89a3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:06:58.339902 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.339869 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83-kube-api-access-x4bk4" (OuterVolumeSpecName: "kube-api-access-x4bk4") pod "2eb20b1e-ee6d-4c72-a079-b6f95edc1e83" (UID: "2eb20b1e-ee6d-4c72-a079-b6f95edc1e83"). InnerVolumeSpecName "kube-api-access-x4bk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:06:58.339902 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.339890 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "41e6892a-633d-49c7-9a90-1d23b5c89a3a" (UID: "41e6892a-633d-49c7-9a90-1d23b5c89a3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:06:58.340049 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.339909 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e6892a-633d-49c7-9a90-1d23b5c89a3a-kube-api-access-2gdcj" (OuterVolumeSpecName: "kube-api-access-2gdcj") pod "41e6892a-633d-49c7-9a90-1d23b5c89a3a" (UID: "41e6892a-633d-49c7-9a90-1d23b5c89a3a"). InnerVolumeSpecName "kube-api-access-2gdcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:06:58.340049 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.339916 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "41e6892a-633d-49c7-9a90-1d23b5c89a3a" (UID: "41e6892a-633d-49c7-9a90-1d23b5c89a3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:06:58.439096 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.439002 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gdcj\" (UniqueName: \"kubernetes.io/projected/41e6892a-633d-49c7-9a90-1d23b5c89a3a-kube-api-access-2gdcj\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:06:58.439096 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.439041 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-maas-api-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:06:58.439096 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.439052 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e6892a-633d-49c7-9a90-1d23b5c89a3a-openshift-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:06:58.439096 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.439060 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4bk4\" (UniqueName: \"kubernetes.io/projected/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83-kube-api-access-x4bk4\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:06:58.439096 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.439069 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e6892a-633d-49c7-9a90-1d23b5c89a3a-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:06:58.439096 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.439077 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:06:58.696765 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.696671 2578 generic.go:356] "Generic (PLEG): container finished" podID="2eb20b1e-ee6d-4c72-a079-b6f95edc1e83" containerID="3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936" exitCode=0 Sep 05 20:06:58.696765 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.696715 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" Sep 05 20:06:58.696988 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.696758 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" event={"ID":"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83","Type":"ContainerDied","Data":"3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936"} Sep 05 20:06:58.696988 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.696845 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2" event={"ID":"2eb20b1e-ee6d-4c72-a079-b6f95edc1e83","Type":"ContainerDied","Data":"1dd53badf58f478931a57832b3708b46a30d5c7959c8cad7e5fe4123574f9fe4"} Sep 05 20:06:58.696988 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.696875 2578 scope.go:117] "RemoveContainer" containerID="3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936" Sep 05 20:06:58.698188 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.698165 2578 generic.go:356] "Generic (PLEG): container finished" podID="41e6892a-633d-49c7-9a90-1d23b5c89a3a" containerID="48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1" exitCode=0 Sep 05 20:06:58.698255 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.698212 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" Sep 05 20:06:58.698308 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.698209 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" event={"ID":"41e6892a-633d-49c7-9a90-1d23b5c89a3a","Type":"ContainerDied","Data":"48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1"} Sep 05 20:06:58.698374 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.698328 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm" event={"ID":"41e6892a-633d-49c7-9a90-1d23b5c89a3a","Type":"ContainerDied","Data":"bae243315074d16ba8f29f7e099ba79d6b3951e75d8f6318acccea75d7aff387"} Sep 05 20:06:58.709257 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.709151 2578 scope.go:117] "RemoveContainer" containerID="3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936" Sep 05 20:06:58.709872 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:06:58.709843 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936\": container with ID starting with 3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936 not found: ID does not exist" containerID="3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936" Sep 05 20:06:58.709980 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.709884 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936"} err="failed to get container status \"3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936\": rpc error: code = NotFound desc = could not find container \"3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936\": container with ID starting with 3fa857239d9d587e768d66bce5071c5671f89489226e1d0943712638f49bc936 not found: ID does not exist" Sep 05 20:06:58.709980 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.709909 2578 scope.go:117] "RemoveContainer" containerID="48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1" Sep 05 20:06:58.718594 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.718576 2578 scope.go:117] "RemoveContainer" containerID="48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1" Sep 05 20:06:58.718891 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:06:58.718872 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1\": container with ID starting with 48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1 not found: ID does not exist" containerID="48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1" Sep 05 20:06:58.718965 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.718904 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1"} err="failed to get container status \"48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1\": rpc error: code = NotFound desc = could not find container \"48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1\": container with ID starting with 48d09eefc081ee397639ba6479cc3d4d53aaad1ab07c061c0b62ec8154fd75e1 not found: ID does not exist" Sep 05 20:06:58.725243 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.725216 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm"] Sep 05 20:06:58.729191 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.729166 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ada83762-dd67dc884-g24gm"] Sep 05 20:06:58.739993 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.739967 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2"] Sep 05 20:06:58.743937 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:06:58.743915 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ada83762-6568f69f45-j8fb2"] Sep 05 20:07:00.012603 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:00.012571 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eb20b1e-ee6d-4c72-a079-b6f95edc1e83" path="/var/lib/kubelet/pods/2eb20b1e-ee6d-4c72-a079-b6f95edc1e83/volumes" Sep 05 20:07:00.013001 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:00.012987 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e6892a-633d-49c7-9a90-1d23b5c89a3a" path="/var/lib/kubelet/pods/41e6892a-633d-49c7-9a90-1d23b5c89a3a/volumes" Sep 05 20:07:11.265015 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.264982 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7845-fd40b4"] Sep 05 20:07:11.265372 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.265361 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb20b1e-ee6d-4c72-a079-b6f95edc1e83" containerName="payload-processing" Sep 05 20:07:11.265372 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.265372 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb20b1e-ee6d-4c72-a079-b6f95edc1e83" containerName="payload-processing" Sep 05 20:07:11.265444 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.265386 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e6892a-633d-49c7-9a90-1d23b5c89a3a" containerName="maas-api" Sep 05 20:07:11.265444 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.265392 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e6892a-633d-49c7-9a90-1d23b5c89a3a" containerName="maas-api" Sep 05 20:07:11.265511 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.265453 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb20b1e-ee6d-4c72-a079-b6f95edc1e83" containerName="payload-processing" Sep 05 20:07:11.265511 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.265462 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e6892a-633d-49c7-9a90-1d23b5c89a3a" containerName="maas-api" Sep 05 20:07:11.269866 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.269848 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7845-fd40b4" Sep 05 20:07:11.272747 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.272694 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cqkks\"" Sep 05 20:07:11.274787 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.274765 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7845-fd40b4"] Sep 05 20:07:11.360500 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.360464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qt8d\" (UniqueName: \"kubernetes.io/projected/60c5a772-6a01-4c1b-9a4a-f6d881dc4c16-kube-api-access-5qt8d\") pod \"mt-curl-7845-fd40b4\" (UID: \"60c5a772-6a01-4c1b-9a4a-f6d881dc4c16\") " pod="opendatahub/mt-curl-7845-fd40b4" Sep 05 20:07:11.461274 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.461226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qt8d\" (UniqueName: \"kubernetes.io/projected/60c5a772-6a01-4c1b-9a4a-f6d881dc4c16-kube-api-access-5qt8d\") pod \"mt-curl-7845-fd40b4\" (UID: \"60c5a772-6a01-4c1b-9a4a-f6d881dc4c16\") " pod="opendatahub/mt-curl-7845-fd40b4" Sep 05 20:07:11.472119 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.472080 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qt8d\" (UniqueName: \"kubernetes.io/projected/60c5a772-6a01-4c1b-9a4a-f6d881dc4c16-kube-api-access-5qt8d\") pod \"mt-curl-7845-fd40b4\" (UID: \"60c5a772-6a01-4c1b-9a4a-f6d881dc4c16\") " pod="opendatahub/mt-curl-7845-fd40b4" Sep 05 20:07:11.580914 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.580815 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7845-fd40b4" Sep 05 20:07:11.852649 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:11.852566 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7845-fd40b4"] Sep 05 20:07:11.856141 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:07:11.856113 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60c5a772_6a01_4c1b_9a4a_f6d881dc4c16.slice/crio-609f1cddedfabf0a9165c40341d406cc9297bea92167a0a40da1dd0dcf42858a WatchSource:0}: Error finding container 609f1cddedfabf0a9165c40341d406cc9297bea92167a0a40da1dd0dcf42858a: Status 404 returned error can't find the container with id 609f1cddedfabf0a9165c40341d406cc9297bea92167a0a40da1dd0dcf42858a Sep 05 20:07:12.768643 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:12.768606 2578 generic.go:356] "Generic (PLEG): container finished" podID="60c5a772-6a01-4c1b-9a4a-f6d881dc4c16" containerID="9cf8eb654fb21f5ca228f87658ae98815b3739c1545c85e944bf2085b1a1ca1e" exitCode=0 Sep 05 20:07:12.769061 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:12.768653 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7845-fd40b4" event={"ID":"60c5a772-6a01-4c1b-9a4a-f6d881dc4c16","Type":"ContainerDied","Data":"9cf8eb654fb21f5ca228f87658ae98815b3739c1545c85e944bf2085b1a1ca1e"} Sep 05 20:07:12.769061 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:12.768696 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7845-fd40b4" event={"ID":"60c5a772-6a01-4c1b-9a4a-f6d881dc4c16","Type":"ContainerStarted","Data":"609f1cddedfabf0a9165c40341d406cc9297bea92167a0a40da1dd0dcf42858a"} Sep 05 20:07:13.916247 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:13.916216 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7845-fd40b4" Sep 05 20:07:13.950096 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:13.950067 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7845-fd40b4_60c5a772-6a01-4c1b-9a4a-f6d881dc4c16/mt-curl-7845-fd40b4/0.log" Sep 05 20:07:13.972189 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:13.972142 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7845-fd40b4"] Sep 05 20:07:13.975601 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:13.975572 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7845-fd40b4"] Sep 05 20:07:13.983127 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:13.983103 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qt8d\" (UniqueName: \"kubernetes.io/projected/60c5a772-6a01-4c1b-9a4a-f6d881dc4c16-kube-api-access-5qt8d\") pod \"60c5a772-6a01-4c1b-9a4a-f6d881dc4c16\" (UID: \"60c5a772-6a01-4c1b-9a4a-f6d881dc4c16\") " Sep 05 20:07:13.985143 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:13.985118 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c5a772-6a01-4c1b-9a4a-f6d881dc4c16-kube-api-access-5qt8d" (OuterVolumeSpecName: "kube-api-access-5qt8d") pod "60c5a772-6a01-4c1b-9a4a-f6d881dc4c16" (UID: "60c5a772-6a01-4c1b-9a4a-f6d881dc4c16"). InnerVolumeSpecName "kube-api-access-5qt8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:07:14.012486 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:14.012454 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60c5a772-6a01-4c1b-9a4a-f6d881dc4c16" path="/var/lib/kubelet/pods/60c5a772-6a01-4c1b-9a4a-f6d881dc4c16/volumes" Sep 05 20:07:14.084869 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:14.084768 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qt8d\" (UniqueName: \"kubernetes.io/projected/60c5a772-6a01-4c1b-9a4a-f6d881dc4c16-kube-api-access-5qt8d\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:07:14.780054 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:14.780023 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7845-fd40b4" Sep 05 20:07:14.780247 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:07:14.780030 2578 scope.go:117] "RemoveContainer" containerID="9cf8eb654fb21f5ca228f87658ae98815b3739c1545c85e944bf2085b1a1ca1e" Sep 05 20:08:00.064967 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:00.064938 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 20:08:00.071673 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:00.071651 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 20:08:49.485761 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:49.485716 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7845-90ed61"] Sep 05 20:08:49.486372 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:49.486349 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c5a772-6a01-4c1b-9a4a-f6d881dc4c16" containerName="mt-curl-7845-fd40b4" Sep 05 20:08:49.486455 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:49.486374 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c5a772-6a01-4c1b-9a4a-f6d881dc4c16" containerName="mt-curl-7845-fd40b4" Sep 05 20:08:49.486512 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:49.486477 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c5a772-6a01-4c1b-9a4a-f6d881dc4c16" containerName="mt-curl-7845-fd40b4" Sep 05 20:08:49.489584 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:49.489564 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7845-90ed61" Sep 05 20:08:49.492292 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:49.492266 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cqkks\"" Sep 05 20:08:49.495965 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:49.495918 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7845-90ed61"] Sep 05 20:08:49.567681 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:49.567634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5n2v\" (UniqueName: \"kubernetes.io/projected/61c4ab39-e18f-4c74-a25f-8cfa389fda2c-kube-api-access-v5n2v\") pod \"mt-curl-7845-90ed61\" (UID: \"61c4ab39-e18f-4c74-a25f-8cfa389fda2c\") " pod="opendatahub/mt-curl-7845-90ed61" Sep 05 20:08:49.669153 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:49.669117 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5n2v\" (UniqueName: \"kubernetes.io/projected/61c4ab39-e18f-4c74-a25f-8cfa389fda2c-kube-api-access-v5n2v\") pod \"mt-curl-7845-90ed61\" (UID: \"61c4ab39-e18f-4c74-a25f-8cfa389fda2c\") " pod="opendatahub/mt-curl-7845-90ed61" Sep 05 20:08:49.677940 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:49.677910 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5n2v\" (UniqueName: \"kubernetes.io/projected/61c4ab39-e18f-4c74-a25f-8cfa389fda2c-kube-api-access-v5n2v\") pod \"mt-curl-7845-90ed61\" (UID: \"61c4ab39-e18f-4c74-a25f-8cfa389fda2c\") " pod="opendatahub/mt-curl-7845-90ed61" Sep 05 20:08:49.799661 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:49.799619 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7845-90ed61" Sep 05 20:08:50.070521 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.070494 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7845-90ed61"] Sep 05 20:08:50.072625 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:08:50.072588 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61c4ab39_e18f_4c74_a25f_8cfa389fda2c.slice/crio-97dd7f07482abfc7a363eef8acf89fd68ea5a0957a0579a3ee303bf9fdef68a4 WatchSource:0}: Error finding container 97dd7f07482abfc7a363eef8acf89fd68ea5a0957a0579a3ee303bf9fdef68a4: Status 404 returned error can't find the container with id 97dd7f07482abfc7a363eef8acf89fd68ea5a0957a0579a3ee303bf9fdef68a4 Sep 05 20:08:50.074018 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.073998 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 20:08:50.269065 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.269029 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7845-90ed61" event={"ID":"61c4ab39-e18f-4c74-a25f-8cfa389fda2c","Type":"ContainerStarted","Data":"97dd7f07482abfc7a363eef8acf89fd68ea5a0957a0579a3ee303bf9fdef68a4"} Sep 05 20:08:50.415699 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.415671 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p"] Sep 05 20:08:50.419189 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.419165 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:50.422166 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.422002 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1aa465-serving-cert\"" Sep 05 20:08:50.429946 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.429824 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p"] Sep 05 20:08:50.577022 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.576978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-metrics-certs\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:50.577434 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.577055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsb6d\" (UniqueName: \"kubernetes.io/projected/64230d0d-9fea-4cac-ac29-584c459b4b67-kube-api-access-lsb6d\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:50.577434 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.577108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64230d0d-9fea-4cac-ac29-584c459b4b67-openshift-service-ca\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:50.577434 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.577188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-maas-api-tls\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:50.678316 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.678226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-metrics-certs\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:50.678316 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.678280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsb6d\" (UniqueName: \"kubernetes.io/projected/64230d0d-9fea-4cac-ac29-584c459b4b67-kube-api-access-lsb6d\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:50.678316 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.678316 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64230d0d-9fea-4cac-ac29-584c459b4b67-openshift-service-ca\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:50.678608 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.678347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-maas-api-tls\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:50.678608 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:08:50.678471 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-serving-cert: secret "maas-api-e2e-api-a-1aa465-serving-cert" not found Sep 05 20:08:50.678608 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:08:50.678530 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-maas-api-tls podName:64230d0d-9fea-4cac-ac29-584c459b4b67 nodeName:}" failed. No retries permitted until 2026-09-05 20:08:51.178510956 +0000 UTC m=+1551.663811821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-maas-api-tls") pod "maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" (UID: "64230d0d-9fea-4cac-ac29-584c459b4b67") : secret "maas-api-e2e-api-a-1aa465-serving-cert" not found Sep 05 20:08:50.679099 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.679077 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64230d0d-9fea-4cac-ac29-584c459b4b67-openshift-service-ca\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:50.680824 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.680773 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-metrics-certs\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:50.689864 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:50.689842 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsb6d\" (UniqueName: \"kubernetes.io/projected/64230d0d-9fea-4cac-ac29-584c459b4b67-kube-api-access-lsb6d\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:51.183485 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:51.183451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-maas-api-tls\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:51.183668 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:08:51.183611 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-serving-cert: secret "maas-api-e2e-api-a-1aa465-serving-cert" not found Sep 05 20:08:51.183713 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:08:51.183676 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-maas-api-tls podName:64230d0d-9fea-4cac-ac29-584c459b4b67 nodeName:}" failed. No retries permitted until 2026-09-05 20:08:52.183658716 +0000 UTC m=+1552.668959585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-maas-api-tls") pod "maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" (UID: "64230d0d-9fea-4cac-ac29-584c459b4b67") : secret "maas-api-e2e-api-a-1aa465-serving-cert" not found Sep 05 20:08:51.275911 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:51.275871 2578 generic.go:356] "Generic (PLEG): container finished" podID="61c4ab39-e18f-4c74-a25f-8cfa389fda2c" containerID="5191027716cbd5b419d4da94e62c628e0324121371df7e1c3a314266e984c746" exitCode=0 Sep 05 20:08:51.276109 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:51.275958 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7845-90ed61" event={"ID":"61c4ab39-e18f-4c74-a25f-8cfa389fda2c","Type":"ContainerDied","Data":"5191027716cbd5b419d4da94e62c628e0324121371df7e1c3a314266e984c746"} Sep 05 20:08:52.194695 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:52.194655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-maas-api-tls\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:52.197147 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:52.197120 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-maas-api-tls\") pod \"maas-api-e2e-api-a-1aa465-bb94676dc-krp9p\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:52.232921 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:52.232878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:52.429373 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:52.429340 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7845-90ed61" Sep 05 20:08:52.462574 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:52.462496 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7845-90ed61_61c4ab39-e18f-4c74-a25f-8cfa389fda2c/mt-curl-7845-90ed61/0.log" Sep 05 20:08:52.481818 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:52.481760 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7845-90ed61"] Sep 05 20:08:52.488373 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:52.488345 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7845-90ed61"] Sep 05 20:08:52.508278 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:52.508255 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p"] Sep 05 20:08:52.509792 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:08:52.509762 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64230d0d_9fea_4cac_ac29_584c459b4b67.slice/crio-1b3579483923db70930792a149c7863eb7aeba559f54298bf8cda251d3909ac2 WatchSource:0}: Error finding container 1b3579483923db70930792a149c7863eb7aeba559f54298bf8cda251d3909ac2: Status 404 returned error can't find the container with id 1b3579483923db70930792a149c7863eb7aeba559f54298bf8cda251d3909ac2 Sep 05 20:08:52.598425 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:52.598391 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5n2v\" (UniqueName: \"kubernetes.io/projected/61c4ab39-e18f-4c74-a25f-8cfa389fda2c-kube-api-access-v5n2v\") pod \"61c4ab39-e18f-4c74-a25f-8cfa389fda2c\" (UID: \"61c4ab39-e18f-4c74-a25f-8cfa389fda2c\") " Sep 05 20:08:52.600572 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:52.600544 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c4ab39-e18f-4c74-a25f-8cfa389fda2c-kube-api-access-v5n2v" (OuterVolumeSpecName: "kube-api-access-v5n2v") pod "61c4ab39-e18f-4c74-a25f-8cfa389fda2c" (UID: "61c4ab39-e18f-4c74-a25f-8cfa389fda2c"). InnerVolumeSpecName "kube-api-access-v5n2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:08:52.699335 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:52.699296 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5n2v\" (UniqueName: \"kubernetes.io/projected/61c4ab39-e18f-4c74-a25f-8cfa389fda2c-kube-api-access-v5n2v\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:08:53.289863 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:53.289822 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" event={"ID":"64230d0d-9fea-4cac-ac29-584c459b4b67","Type":"ContainerStarted","Data":"aa33deb37314bd9eb2cb6be658a3a08bb80a4da9c4c031af4ca8cae5dd5e853a"} Sep 05 20:08:53.290310 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:53.289884 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" event={"ID":"64230d0d-9fea-4cac-ac29-584c459b4b67","Type":"ContainerStarted","Data":"1b3579483923db70930792a149c7863eb7aeba559f54298bf8cda251d3909ac2"} Sep 05 20:08:53.290310 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:53.289987 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:53.295845 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:53.294425 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97dd7f07482abfc7a363eef8acf89fd68ea5a0957a0579a3ee303bf9fdef68a4" Sep 05 20:08:53.295845 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:53.294503 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7845-90ed61" Sep 05 20:08:53.313773 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:53.313718 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" podStartSLOduration=2.902002042 podStartE2EDuration="3.313702819s" totalImagesPullingTime="411.700777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:08:53.313511224 +0000 UTC m=+1553.798812100" watchObservedRunningTime="2026-09-05 20:08:53.313702819 +0000 UTC m=+1553.799003691" Sep 05 20:08:53.315912 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:08:53.315882 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7845-90ed61\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="61c4ab39-e18f-4c74-a25f-8cfa389fda2c" pod="opendatahub/mt-curl-7845-90ed61" Sep 05 20:08:54.013364 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:54.013329 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61c4ab39-e18f-4c74-a25f-8cfa389fda2c" path="/var/lib/kubelet/pods/61c4ab39-e18f-4c74-a25f-8cfa389fda2c/volumes" Sep 05 20:08:58.234010 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.233973 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94"] Sep 05 20:08:58.234377 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.234362 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61c4ab39-e18f-4c74-a25f-8cfa389fda2c" containerName="mt-curl-7845-90ed61" Sep 05 20:08:58.234422 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.234378 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="61c4ab39-e18f-4c74-a25f-8cfa389fda2c" containerName="mt-curl-7845-90ed61" Sep 05 20:08:58.234461 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.234443 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="61c4ab39-e18f-4c74-a25f-8cfa389fda2c" containerName="mt-curl-7845-90ed61" Sep 05 20:08:58.237780 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.237762 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.242003 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.241974 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4d1885\"/\"rate-test-model-4d1885-kserve-self-signed-certs\"" Sep 05 20:08:58.242003 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.241996 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4d1885\"/\"openshift-service-ca.crt\"" Sep 05 20:08:58.242182 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.241996 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4d1885\"/\"kube-root-ca.crt\"" Sep 05 20:08:58.242182 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.241987 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4d1885\"/\"default-dockercfg-hrt2t\"" Sep 05 20:08:58.247055 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.247019 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94"] Sep 05 20:08:58.358748 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.358712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-tmp-dir\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.358947 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.358755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-model-cache\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.358947 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.358782 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-dshm\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.358947 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.358911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99svg\" (UniqueName: \"kubernetes.io/projected/ff83810f-faec-4a31-b7e3-3f8e8872c92d-kube-api-access-99svg\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.359067 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.358971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-home\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.359067 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.359011 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff83810f-faec-4a31-b7e3-3f8e8872c92d-tls-certs\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.460331 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.460288 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-dshm\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.460513 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.460362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99svg\" (UniqueName: \"kubernetes.io/projected/ff83810f-faec-4a31-b7e3-3f8e8872c92d-kube-api-access-99svg\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.460513 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.460405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-home\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.460513 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.460440 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff83810f-faec-4a31-b7e3-3f8e8872c92d-tls-certs\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.460615 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.460572 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-tmp-dir\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.460660 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.460647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-model-cache\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.460853 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.460821 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-home\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.460996 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.460942 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-tmp-dir\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.461055 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.460995 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-model-cache\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.462655 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.462630 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-dshm\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.462986 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.462969 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff83810f-faec-4a31-b7e3-3f8e8872c92d-tls-certs\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.470939 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.470914 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99svg\" (UniqueName: \"kubernetes.io/projected/ff83810f-faec-4a31-b7e3-3f8e8872c92d-kube-api-access-99svg\") pod \"rate-test-model-4d1885-kserve-567bc65579-stj94\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.547526 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.547486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:08:58.827228 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:58.827164 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94"] Sep 05 20:08:58.829510 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:08:58.829483 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff83810f_faec_4a31_b7e3_3f8e8872c92d.slice/crio-5c96d840ead413e9e35ba348b15d70132f3a1eb40e293aec040d6c4934f3970d WatchSource:0}: Error finding container 5c96d840ead413e9e35ba348b15d70132f3a1eb40e293aec040d6c4934f3970d: Status 404 returned error can't find the container with id 5c96d840ead413e9e35ba348b15d70132f3a1eb40e293aec040d6c4934f3970d Sep 05 20:08:59.304718 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:59.304685 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:08:59.331986 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:59.331942 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" event={"ID":"ff83810f-faec-4a31-b7e3-3f8e8872c92d","Type":"ContainerStarted","Data":"48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7"} Sep 05 20:08:59.332187 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:59.332001 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" event={"ID":"ff83810f-faec-4a31-b7e3-3f8e8872c92d","Type":"ContainerStarted","Data":"5c96d840ead413e9e35ba348b15d70132f3a1eb40e293aec040d6c4934f3970d"} Sep 05 20:08:59.354487 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:08:59.354421 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" podStartSLOduration=1.354405304 podStartE2EDuration="1.354405304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:08:59.351258884 +0000 UTC m=+1559.836559780" watchObservedRunningTime="2026-09-05 20:08:59.354405304 +0000 UTC m=+1559.839706177" Sep 05 20:09:00.176084 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:00.176050 2578 scope.go:117] "RemoveContainer" containerID="59d255682b6c63252817332b913fd6f0cbd546e4e15dbdc3585d9f3346bbe593" Sep 05 20:09:08.547746 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:08.547710 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:09:08.548230 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:08.547766 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:09:08.560460 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:08.560432 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:09:09.393982 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:09.393953 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:09:20.116703 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.116664 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj"] Sep 05 20:09:20.120449 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.120420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" Sep 05 20:09:20.123446 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.123421 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-dbbedd-dockercfg-vxrqf\"" Sep 05 20:09:20.123581 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.123528 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-dbbedd\"" Sep 05 20:09:20.133820 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.133774 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj"] Sep 05 20:09:20.279195 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.279158 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlq48\" (UniqueName: \"kubernetes.io/projected/fd5f38f6-e436-4cff-9f89-e66a978643f1-kube-api-access-tlq48\") pod \"payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj\" (UID: \"fd5f38f6-e436-4cff-9f89-e66a978643f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" Sep 05 20:09:20.279365 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.279289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5f38f6-e436-4cff-9f89-e66a978643f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj\" (UID: \"fd5f38f6-e436-4cff-9f89-e66a978643f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" Sep 05 20:09:20.380093 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.379988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5f38f6-e436-4cff-9f89-e66a978643f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj\" (UID: \"fd5f38f6-e436-4cff-9f89-e66a978643f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" Sep 05 20:09:20.380093 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.380062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlq48\" (UniqueName: \"kubernetes.io/projected/fd5f38f6-e436-4cff-9f89-e66a978643f1-kube-api-access-tlq48\") pod \"payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj\" (UID: \"fd5f38f6-e436-4cff-9f89-e66a978643f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" Sep 05 20:09:20.380826 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.380786 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5f38f6-e436-4cff-9f89-e66a978643f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj\" (UID: \"fd5f38f6-e436-4cff-9f89-e66a978643f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" Sep 05 20:09:20.394031 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.394002 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlq48\" (UniqueName: \"kubernetes.io/projected/fd5f38f6-e436-4cff-9f89-e66a978643f1-kube-api-access-tlq48\") pod \"payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj\" (UID: \"fd5f38f6-e436-4cff-9f89-e66a978643f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" Sep 05 20:09:20.431770 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.431729 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" Sep 05 20:09:20.706196 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:20.706171 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj"] Sep 05 20:09:20.708608 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:09:20.708575 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5f38f6_e436_4cff_9f89_e66a978643f1.slice/crio-8505bfe0b716ceda16917405078d98416ae3cb3041feb6ca9b975739d27fde34 WatchSource:0}: Error finding container 8505bfe0b716ceda16917405078d98416ae3cb3041feb6ca9b975739d27fde34: Status 404 returned error can't find the container with id 8505bfe0b716ceda16917405078d98416ae3cb3041feb6ca9b975739d27fde34 Sep 05 20:09:21.446297 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:21.446253 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" event={"ID":"fd5f38f6-e436-4cff-9f89-e66a978643f1","Type":"ContainerStarted","Data":"04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10"} Sep 05 20:09:21.446763 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:21.446324 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" event={"ID":"fd5f38f6-e436-4cff-9f89-e66a978643f1","Type":"ContainerStarted","Data":"8505bfe0b716ceda16917405078d98416ae3cb3041feb6ca9b975739d27fde34"} Sep 05 20:09:21.446763 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:21.446351 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" Sep 05 20:09:21.465309 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:21.465267 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" podStartSLOduration=1.465251656 podStartE2EDuration="1.465251656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:09:21.464507552 +0000 UTC m=+1581.949808426" watchObservedRunningTime="2026-09-05 20:09:21.465251656 +0000 UTC m=+1581.950552562" Sep 05 20:09:32.454844 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:09:32.454775 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" Sep 05 20:10:01.464271 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.464174 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p"] Sep 05 20:10:01.464867 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.464484 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" podUID="64230d0d-9fea-4cac-ac29-584c459b4b67" containerName="maas-api" containerID="cri-o://aa33deb37314bd9eb2cb6be658a3a08bb80a4da9c4c031af4ca8cae5dd5e853a" gracePeriod=30 Sep 05 20:10:01.652205 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.652170 2578 generic.go:356] "Generic (PLEG): container finished" podID="64230d0d-9fea-4cac-ac29-584c459b4b67" containerID="aa33deb37314bd9eb2cb6be658a3a08bb80a4da9c4c031af4ca8cae5dd5e853a" exitCode=0 Sep 05 20:10:01.652368 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.652254 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" event={"ID":"64230d0d-9fea-4cac-ac29-584c459b4b67","Type":"ContainerDied","Data":"aa33deb37314bd9eb2cb6be658a3a08bb80a4da9c4c031af4ca8cae5dd5e853a"} Sep 05 20:10:01.719447 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.719374 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:10:01.892100 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.892060 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-metrics-certs\") pod \"64230d0d-9fea-4cac-ac29-584c459b4b67\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " Sep 05 20:10:01.892279 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.892130 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsb6d\" (UniqueName: \"kubernetes.io/projected/64230d0d-9fea-4cac-ac29-584c459b4b67-kube-api-access-lsb6d\") pod \"64230d0d-9fea-4cac-ac29-584c459b4b67\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " Sep 05 20:10:01.892279 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.892191 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64230d0d-9fea-4cac-ac29-584c459b4b67-openshift-service-ca\") pod \"64230d0d-9fea-4cac-ac29-584c459b4b67\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " Sep 05 20:10:01.892279 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.892249 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-maas-api-tls\") pod \"64230d0d-9fea-4cac-ac29-584c459b4b67\" (UID: \"64230d0d-9fea-4cac-ac29-584c459b4b67\") " Sep 05 20:10:01.892626 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.892594 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64230d0d-9fea-4cac-ac29-584c459b4b67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "64230d0d-9fea-4cac-ac29-584c459b4b67" (UID: "64230d0d-9fea-4cac-ac29-584c459b4b67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:10:01.894303 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.894276 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "64230d0d-9fea-4cac-ac29-584c459b4b67" (UID: "64230d0d-9fea-4cac-ac29-584c459b4b67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:10:01.894690 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.894666 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "64230d0d-9fea-4cac-ac29-584c459b4b67" (UID: "64230d0d-9fea-4cac-ac29-584c459b4b67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:10:01.894766 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.894685 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64230d0d-9fea-4cac-ac29-584c459b4b67-kube-api-access-lsb6d" (OuterVolumeSpecName: "kube-api-access-lsb6d") pod "64230d0d-9fea-4cac-ac29-584c459b4b67" (UID: "64230d0d-9fea-4cac-ac29-584c459b4b67"). InnerVolumeSpecName "kube-api-access-lsb6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:10:01.994078 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.993996 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-maas-api-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:01.994078 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.994026 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64230d0d-9fea-4cac-ac29-584c459b4b67-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:01.994078 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.994035 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsb6d\" (UniqueName: \"kubernetes.io/projected/64230d0d-9fea-4cac-ac29-584c459b4b67-kube-api-access-lsb6d\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:01.994078 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:01.994044 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64230d0d-9fea-4cac-ac29-584c459b4b67-openshift-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:02.658918 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:02.658893 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" Sep 05 20:10:02.659331 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:02.658889 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p" event={"ID":"64230d0d-9fea-4cac-ac29-584c459b4b67","Type":"ContainerDied","Data":"1b3579483923db70930792a149c7863eb7aeba559f54298bf8cda251d3909ac2"} Sep 05 20:10:02.659331 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:02.659031 2578 scope.go:117] "RemoveContainer" containerID="aa33deb37314bd9eb2cb6be658a3a08bb80a4da9c4c031af4ca8cae5dd5e853a" Sep 05 20:10:02.682163 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:02.682129 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p"] Sep 05 20:10:02.686437 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:02.686410 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1aa465-bb94676dc-krp9p"] Sep 05 20:10:04.013202 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:04.013166 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64230d0d-9fea-4cac-ac29-584c459b4b67" path="/var/lib/kubelet/pods/64230d0d-9fea-4cac-ac29-584c459b4b67/volumes" Sep 05 20:10:24.337611 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.337132 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj"] Sep 05 20:10:24.337611 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.337399 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" podUID="fd5f38f6-e436-4cff-9f89-e66a978643f1" containerName="payload-pre-processing" containerID="cri-o://04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10" gracePeriod=30 Sep 05 20:10:24.592816 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.592716 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" Sep 05 20:10:24.692906 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.692869 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5f38f6-e436-4cff-9f89-e66a978643f1-plugins-config-volume\") pod \"fd5f38f6-e436-4cff-9f89-e66a978643f1\" (UID: \"fd5f38f6-e436-4cff-9f89-e66a978643f1\") " Sep 05 20:10:24.693100 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.693076 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlq48\" (UniqueName: \"kubernetes.io/projected/fd5f38f6-e436-4cff-9f89-e66a978643f1-kube-api-access-tlq48\") pod \"fd5f38f6-e436-4cff-9f89-e66a978643f1\" (UID: \"fd5f38f6-e436-4cff-9f89-e66a978643f1\") " Sep 05 20:10:24.693348 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.693320 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5f38f6-e436-4cff-9f89-e66a978643f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd5f38f6-e436-4cff-9f89-e66a978643f1" (UID: "fd5f38f6-e436-4cff-9f89-e66a978643f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:10:24.693436 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.693388 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5f38f6-e436-4cff-9f89-e66a978643f1-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:24.695156 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.695135 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5f38f6-e436-4cff-9f89-e66a978643f1-kube-api-access-tlq48" (OuterVolumeSpecName: "kube-api-access-tlq48") pod "fd5f38f6-e436-4cff-9f89-e66a978643f1" (UID: "fd5f38f6-e436-4cff-9f89-e66a978643f1"). InnerVolumeSpecName "kube-api-access-tlq48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:10:24.773490 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.773457 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd5f38f6-e436-4cff-9f89-e66a978643f1" containerID="04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10" exitCode=0 Sep 05 20:10:24.773652 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.773501 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" Sep 05 20:10:24.773652 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.773539 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" event={"ID":"fd5f38f6-e436-4cff-9f89-e66a978643f1","Type":"ContainerDied","Data":"04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10"} Sep 05 20:10:24.773652 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.773628 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj" event={"ID":"fd5f38f6-e436-4cff-9f89-e66a978643f1","Type":"ContainerDied","Data":"8505bfe0b716ceda16917405078d98416ae3cb3041feb6ca9b975739d27fde34"} Sep 05 20:10:24.773652 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.773652 2578 scope.go:117] "RemoveContainer" containerID="04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10" Sep 05 20:10:24.782844 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.782824 2578 scope.go:117] "RemoveContainer" containerID="04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10" Sep 05 20:10:24.783147 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:10:24.783126 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10\": container with ID starting with 04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10 not found: ID does not exist" containerID="04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10" Sep 05 20:10:24.783221 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.783158 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10"} err="failed to get container status \"04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10\": rpc error: code = NotFound desc = could not find container \"04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10\": container with ID starting with 04e6d2abe100aa2c604537388745044acf5e5793b6d5d62b5a8ebe3fcbbfeb10 not found: ID does not exist" Sep 05 20:10:24.794611 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.794578 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlq48\" (UniqueName: \"kubernetes.io/projected/fd5f38f6-e436-4cff-9f89-e66a978643f1-kube-api-access-tlq48\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:24.796719 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.796696 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj"] Sep 05 20:10:24.804086 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:24.804059 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dbbedd-748899d8f4-cvzxj"] Sep 05 20:10:26.012051 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:26.012011 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd5f38f6-e436-4cff-9f89-e66a978643f1" path="/var/lib/kubelet/pods/fd5f38f6-e436-4cff-9f89-e66a978643f1/volumes" Sep 05 20:10:35.221263 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:35.221224 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94"] Sep 05 20:10:41.070947 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.070906 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk"] Sep 05 20:10:41.071393 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.071376 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64230d0d-9fea-4cac-ac29-584c459b4b67" containerName="maas-api" Sep 05 20:10:41.071393 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.071395 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="64230d0d-9fea-4cac-ac29-584c459b4b67" containerName="maas-api" Sep 05 20:10:41.071475 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.071408 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd5f38f6-e436-4cff-9f89-e66a978643f1" containerName="payload-pre-processing" Sep 05 20:10:41.071475 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.071413 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd5f38f6-e436-4cff-9f89-e66a978643f1" containerName="payload-pre-processing" Sep 05 20:10:41.071550 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.071493 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="64230d0d-9fea-4cac-ac29-584c459b4b67" containerName="maas-api" Sep 05 20:10:41.071550 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.071506 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd5f38f6-e436-4cff-9f89-e66a978643f1" containerName="payload-pre-processing" Sep 05 20:10:41.074614 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.074595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.078991 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.078969 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d0f6b5-serving-cert\"" Sep 05 20:10:41.090227 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.090198 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk"] Sep 05 20:10:41.154828 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.154752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b7ba4b-8f12-4548-91db-93daa7e60386-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.155010 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.154913 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b7ba4b-8f12-4548-91db-93daa7e60386-metrics-certs\") pod \"maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.155010 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.154949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgxwv\" (UniqueName: \"kubernetes.io/projected/55b7ba4b-8f12-4548-91db-93daa7e60386-kube-api-access-cgxwv\") pod \"maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.155111 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.155094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b7ba4b-8f12-4548-91db-93daa7e60386-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.256549 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.256509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b7ba4b-8f12-4548-91db-93daa7e60386-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.256721 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.256563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b7ba4b-8f12-4548-91db-93daa7e60386-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.256721 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.256599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b7ba4b-8f12-4548-91db-93daa7e60386-metrics-certs\") pod \"maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.256721 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.256618 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgxwv\" (UniqueName: \"kubernetes.io/projected/55b7ba4b-8f12-4548-91db-93daa7e60386-kube-api-access-cgxwv\") pod \"maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.257267 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.257240 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b7ba4b-8f12-4548-91db-93daa7e60386-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.259205 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.259178 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b7ba4b-8f12-4548-91db-93daa7e60386-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.259306 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.259178 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b7ba4b-8f12-4548-91db-93daa7e60386-metrics-certs\") pod \"maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.265073 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.265044 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgxwv\" (UniqueName: \"kubernetes.io/projected/55b7ba4b-8f12-4548-91db-93daa7e60386-kube-api-access-cgxwv\") pod \"maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.384766 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.384669 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:41.661843 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.661786 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk"] Sep 05 20:10:41.664629 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:10:41.664601 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b7ba4b_8f12_4548_91db_93daa7e60386.slice/crio-923ef6f63d2ec111da0cd247badcb681b60b7ad208be96e91c5832aadf73e43a WatchSource:0}: Error finding container 923ef6f63d2ec111da0cd247badcb681b60b7ad208be96e91c5832aadf73e43a: Status 404 returned error can't find the container with id 923ef6f63d2ec111da0cd247badcb681b60b7ad208be96e91c5832aadf73e43a Sep 05 20:10:41.870212 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:41.870177 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" event={"ID":"55b7ba4b-8f12-4548-91db-93daa7e60386","Type":"ContainerStarted","Data":"923ef6f63d2ec111da0cd247badcb681b60b7ad208be96e91c5832aadf73e43a"} Sep 05 20:10:42.404195 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.404148 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8"] Sep 05 20:10:42.404609 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.404452 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" podUID="6f8d709b-3a6a-4a69-95eb-c44fe8125c3a" containerName="payload-processing" containerID="cri-o://c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2" gracePeriod=30 Sep 05 20:10:42.664981 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.664920 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" Sep 05 20:10:42.772915 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.772871 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg9n9\" (UniqueName: \"kubernetes.io/projected/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a-kube-api-access-dg9n9\") pod \"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a\" (UID: \"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a\") " Sep 05 20:10:42.773125 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.773012 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a-plugins-config-volume\") pod \"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a\" (UID: \"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a\") " Sep 05 20:10:42.773888 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.773853 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f8d709b-3a6a-4a69-95eb-c44fe8125c3a" (UID: "6f8d709b-3a6a-4a69-95eb-c44fe8125c3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:10:42.777695 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.777654 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a-kube-api-access-dg9n9" (OuterVolumeSpecName: "kube-api-access-dg9n9") pod "6f8d709b-3a6a-4a69-95eb-c44fe8125c3a" (UID: "6f8d709b-3a6a-4a69-95eb-c44fe8125c3a"). InnerVolumeSpecName "kube-api-access-dg9n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:10:42.874712 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.874680 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg9n9\" (UniqueName: \"kubernetes.io/projected/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a-kube-api-access-dg9n9\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:42.874712 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.874706 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:42.877144 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.877107 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" event={"ID":"55b7ba4b-8f12-4548-91db-93daa7e60386","Type":"ContainerStarted","Data":"cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625"} Sep 05 20:10:42.877285 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.877210 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:42.878371 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.878348 2578 generic.go:356] "Generic (PLEG): container finished" podID="6f8d709b-3a6a-4a69-95eb-c44fe8125c3a" containerID="c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2" exitCode=0 Sep 05 20:10:42.878477 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.878403 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" Sep 05 20:10:42.878477 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.878423 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" event={"ID":"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a","Type":"ContainerDied","Data":"c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2"} Sep 05 20:10:42.878477 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.878469 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8" event={"ID":"6f8d709b-3a6a-4a69-95eb-c44fe8125c3a","Type":"ContainerDied","Data":"847556c24a98b37d34e119ad4a8edf30c493f1847f6be0d6d372d85e9267e6db"} Sep 05 20:10:42.878630 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.878491 2578 scope.go:117] "RemoveContainer" containerID="c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2" Sep 05 20:10:42.887314 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.887294 2578 scope.go:117] "RemoveContainer" containerID="c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2" Sep 05 20:10:42.887584 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:10:42.887566 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2\": container with ID starting with c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2 not found: ID does not exist" containerID="c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2" Sep 05 20:10:42.887652 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.887591 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2"} err="failed to get container status \"c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2\": rpc error: code = NotFound desc = could not find container \"c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2\": container with ID starting with c9d3d678bfc1d611f6a2196ee6265804d013228064f246362113040033801bb2 not found: ID does not exist" Sep 05 20:10:42.914431 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.914375 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" podStartSLOduration=1.5820342059999999 podStartE2EDuration="1.914358195s" totalImagesPullingTime="332.323989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:10:42.912069415 +0000 UTC m=+1663.397370288" watchObservedRunningTime="2026-09-05 20:10:42.914358195 +0000 UTC m=+1663.399659082" Sep 05 20:10:42.932188 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.932108 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8"] Sep 05 20:10:42.937529 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:42.937498 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1436b9-5bbcdb4cbc-wrlh8"] Sep 05 20:10:44.013513 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:44.013479 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f8d709b-3a6a-4a69-95eb-c44fe8125c3a" path="/var/lib/kubelet/pods/6f8d709b-3a6a-4a69-95eb-c44fe8125c3a/volumes" Sep 05 20:10:48.891231 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:48.891201 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:10:50.241269 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.241230 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" podUID="ff83810f-faec-4a31-b7e3-3f8e8872c92d" containerName="main" containerID="cri-o://48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7" gracePeriod=45 Sep 05 20:10:50.496812 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.496735 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:10:50.649483 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.649450 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99svg\" (UniqueName: \"kubernetes.io/projected/ff83810f-faec-4a31-b7e3-3f8e8872c92d-kube-api-access-99svg\") pod \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " Sep 05 20:10:50.649483 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.649490 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-dshm\") pod \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " Sep 05 20:10:50.649694 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.649539 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-tmp-dir\") pod \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " Sep 05 20:10:50.649694 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.649555 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-model-cache\") pod \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " Sep 05 20:10:50.649694 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.649638 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-home\") pod \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " Sep 05 20:10:50.649694 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.649683 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff83810f-faec-4a31-b7e3-3f8e8872c92d-tls-certs\") pod \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\" (UID: \"ff83810f-faec-4a31-b7e3-3f8e8872c92d\") " Sep 05 20:10:50.649977 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.649897 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff83810f-faec-4a31-b7e3-3f8e8872c92d" (UID: "ff83810f-faec-4a31-b7e3-3f8e8872c92d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:10:50.649977 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.649914 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff83810f-faec-4a31-b7e3-3f8e8872c92d" (UID: "ff83810f-faec-4a31-b7e3-3f8e8872c92d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:10:50.649977 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.649934 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-home" (OuterVolumeSpecName: "home") pod "ff83810f-faec-4a31-b7e3-3f8e8872c92d" (UID: "ff83810f-faec-4a31-b7e3-3f8e8872c92d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:10:50.650138 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.650058 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:50.650138 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.650073 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:50.650138 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.650082 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:50.651996 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.651967 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff83810f-faec-4a31-b7e3-3f8e8872c92d-kube-api-access-99svg" (OuterVolumeSpecName: "kube-api-access-99svg") pod "ff83810f-faec-4a31-b7e3-3f8e8872c92d" (UID: "ff83810f-faec-4a31-b7e3-3f8e8872c92d"). InnerVolumeSpecName "kube-api-access-99svg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:10:50.652202 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.652176 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-dshm" (OuterVolumeSpecName: "dshm") pod "ff83810f-faec-4a31-b7e3-3f8e8872c92d" (UID: "ff83810f-faec-4a31-b7e3-3f8e8872c92d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 20:10:50.652284 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.652171 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff83810f-faec-4a31-b7e3-3f8e8872c92d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff83810f-faec-4a31-b7e3-3f8e8872c92d" (UID: "ff83810f-faec-4a31-b7e3-3f8e8872c92d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:10:50.750998 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.750906 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99svg\" (UniqueName: \"kubernetes.io/projected/ff83810f-faec-4a31-b7e3-3f8e8872c92d-kube-api-access-99svg\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:50.750998 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.750944 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff83810f-faec-4a31-b7e3-3f8e8872c92d-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:50.750998 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.750954 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff83810f-faec-4a31-b7e3-3f8e8872c92d-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:10:50.923489 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.923447 2578 generic.go:356] "Generic (PLEG): container finished" podID="ff83810f-faec-4a31-b7e3-3f8e8872c92d" containerID="48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7" exitCode=0 Sep 05 20:10:50.923489 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.923493 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" Sep 05 20:10:50.923695 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.923530 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" event={"ID":"ff83810f-faec-4a31-b7e3-3f8e8872c92d","Type":"ContainerDied","Data":"48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7"} Sep 05 20:10:50.923695 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.923577 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94" event={"ID":"ff83810f-faec-4a31-b7e3-3f8e8872c92d","Type":"ContainerDied","Data":"5c96d840ead413e9e35ba348b15d70132f3a1eb40e293aec040d6c4934f3970d"} Sep 05 20:10:50.923695 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.923596 2578 scope.go:117] "RemoveContainer" containerID="48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7" Sep 05 20:10:50.933328 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.933308 2578 scope.go:117] "RemoveContainer" containerID="48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7" Sep 05 20:10:50.933591 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:10:50.933570 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7\": container with ID starting with 48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7 not found: ID does not exist" containerID="48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7" Sep 05 20:10:50.933640 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.933599 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7"} err="failed to get container status \"48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7\": rpc error: code = NotFound desc = could not find container \"48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7\": container with ID starting with 48ff0eba889dfd24933407916623f61ba1800f5360cd200f0f355b989bea0ec7 not found: ID does not exist" Sep 05 20:10:50.946107 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.946080 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94"] Sep 05 20:10:50.950558 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:50.950538 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/rate-test-model-4d1885-kserve-567bc65579-stj94"] Sep 05 20:10:52.013306 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:10:52.013266 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff83810f-faec-4a31-b7e3-3f8e8872c92d" path="/var/lib/kubelet/pods/ff83810f-faec-4a31-b7e3-3f8e8872c92d/volumes" Sep 05 20:11:00.200611 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.200572 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl"] Sep 05 20:11:00.201193 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.201156 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" podUID="b0db3130-afc5-4873-89d8-9379c777d9e0" containerName="payload-processing" containerID="cri-o://57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a" gracePeriod=30 Sep 05 20:11:00.459610 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.459536 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" Sep 05 20:11:00.545717 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.545679 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q56s8\" (UniqueName: \"kubernetes.io/projected/b0db3130-afc5-4873-89d8-9379c777d9e0-kube-api-access-q56s8\") pod \"b0db3130-afc5-4873-89d8-9379c777d9e0\" (UID: \"b0db3130-afc5-4873-89d8-9379c777d9e0\") " Sep 05 20:11:00.545912 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.545753 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0db3130-afc5-4873-89d8-9379c777d9e0-plugins-config-volume\") pod \"b0db3130-afc5-4873-89d8-9379c777d9e0\" (UID: \"b0db3130-afc5-4873-89d8-9379c777d9e0\") " Sep 05 20:11:00.546155 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.546132 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0db3130-afc5-4873-89d8-9379c777d9e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0db3130-afc5-4873-89d8-9379c777d9e0" (UID: "b0db3130-afc5-4873-89d8-9379c777d9e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:11:00.547788 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.547765 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0db3130-afc5-4873-89d8-9379c777d9e0-kube-api-access-q56s8" (OuterVolumeSpecName: "kube-api-access-q56s8") pod "b0db3130-afc5-4873-89d8-9379c777d9e0" (UID: "b0db3130-afc5-4873-89d8-9379c777d9e0"). InnerVolumeSpecName "kube-api-access-q56s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:11:00.647403 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.647354 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0db3130-afc5-4873-89d8-9379c777d9e0-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:11:00.647403 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.647393 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q56s8\" (UniqueName: \"kubernetes.io/projected/b0db3130-afc5-4873-89d8-9379c777d9e0-kube-api-access-q56s8\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:11:00.977280 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.977240 2578 generic.go:356] "Generic (PLEG): container finished" podID="b0db3130-afc5-4873-89d8-9379c777d9e0" containerID="57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a" exitCode=0 Sep 05 20:11:00.977463 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.977289 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" Sep 05 20:11:00.977463 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.977308 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" event={"ID":"b0db3130-afc5-4873-89d8-9379c777d9e0","Type":"ContainerDied","Data":"57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a"} Sep 05 20:11:00.977463 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.977358 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl" event={"ID":"b0db3130-afc5-4873-89d8-9379c777d9e0","Type":"ContainerDied","Data":"1aeec4ff1fc69f3ea43735b465dc5d39edb58bf994ee91daa799bc2ea86a4f22"} Sep 05 20:11:00.977463 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.977391 2578 scope.go:117] "RemoveContainer" containerID="57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a" Sep 05 20:11:00.986370 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.986349 2578 scope.go:117] "RemoveContainer" containerID="57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a" Sep 05 20:11:00.986727 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:11:00.986704 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a\": container with ID starting with 57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a not found: ID does not exist" containerID="57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a" Sep 05 20:11:00.986790 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.986737 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a"} err="failed to get container status \"57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a\": rpc error: code = NotFound desc = could not find container \"57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a\": container with ID starting with 57995800c685e6eef61fa6ec72ec87eb540ea0ff438a45dc243a5d3a299ac37a not found: ID does not exist" Sep 05 20:11:00.999439 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:00.999397 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl"] Sep 05 20:11:01.006521 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:01.006491 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca4fba-58df46f549-xc9cl"] Sep 05 20:11:02.013231 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:02.013199 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0db3130-afc5-4873-89d8-9379c777d9e0" path="/var/lib/kubelet/pods/b0db3130-afc5-4873-89d8-9379c777d9e0/volumes" Sep 05 20:11:10.276204 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.276164 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw"] Sep 05 20:11:10.276635 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.276591 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff83810f-faec-4a31-b7e3-3f8e8872c92d" containerName="main" Sep 05 20:11:10.276635 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.276603 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff83810f-faec-4a31-b7e3-3f8e8872c92d" containerName="main" Sep 05 20:11:10.276635 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.276619 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0db3130-afc5-4873-89d8-9379c777d9e0" containerName="payload-processing" Sep 05 20:11:10.276635 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.276625 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0db3130-afc5-4873-89d8-9379c777d9e0" containerName="payload-processing" Sep 05 20:11:10.276784 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.276642 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8d709b-3a6a-4a69-95eb-c44fe8125c3a" containerName="payload-processing" Sep 05 20:11:10.276784 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.276650 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8d709b-3a6a-4a69-95eb-c44fe8125c3a" containerName="payload-processing" Sep 05 20:11:10.276784 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.276714 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f8d709b-3a6a-4a69-95eb-c44fe8125c3a" containerName="payload-processing" Sep 05 20:11:10.276784 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.276722 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0db3130-afc5-4873-89d8-9379c777d9e0" containerName="payload-processing" Sep 05 20:11:10.276784 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.276728 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff83810f-faec-4a31-b7e3-3f8e8872c92d" containerName="main" Sep 05 20:11:10.279962 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.279945 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.283334 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.283307 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6693ae-serving-cert\"" Sep 05 20:11:10.291732 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.291706 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw"] Sep 05 20:11:10.341539 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.341492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfls\" (UniqueName: \"kubernetes.io/projected/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-kube-api-access-brfls\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.341753 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.341548 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.341753 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.341629 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.341753 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.341687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-metrics-certs\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.442650 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.442614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-metrics-certs\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.442650 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.442665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfls\" (UniqueName: \"kubernetes.io/projected/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-kube-api-access-brfls\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.442928 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.442702 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.442928 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.442737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.442928 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:11:10.442884 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-serving-cert: secret "maas-api-e2e-ipp-b-6693ae-serving-cert" not found Sep 05 20:11:10.443096 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:11:10.442960 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-maas-api-tls podName:246ac940-4fd5-40f9-b2e0-d341c1a90e6b nodeName:}" failed. No retries permitted until 2026-09-05 20:11:10.942938045 +0000 UTC m=+1691.428238912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-maas-api-tls") pod "maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" (UID: "246ac940-4fd5-40f9-b2e0-d341c1a90e6b") : secret "maas-api-e2e-ipp-b-6693ae-serving-cert" not found Sep 05 20:11:10.443372 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.443354 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.445109 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.445086 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-metrics-certs\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.452547 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.452523 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfls\" (UniqueName: \"kubernetes.io/projected/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-kube-api-access-brfls\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.948500 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.948461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:10.951191 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:10.951157 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:11.191881 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:11.191841 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:11.470061 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:11.469918 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw"] Sep 05 20:11:11.472966 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:11:11.472936 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246ac940_4fd5_40f9_b2e0_d341c1a90e6b.slice/crio-33c6b253bd19badc368204c96aedb7f9f6c8471431b8a2e0110e335a135d29f1 WatchSource:0}: Error finding container 33c6b253bd19badc368204c96aedb7f9f6c8471431b8a2e0110e335a135d29f1: Status 404 returned error can't find the container with id 33c6b253bd19badc368204c96aedb7f9f6c8471431b8a2e0110e335a135d29f1 Sep 05 20:11:12.034490 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:12.034450 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" event={"ID":"246ac940-4fd5-40f9-b2e0-d341c1a90e6b","Type":"ContainerStarted","Data":"941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d"} Sep 05 20:11:12.034670 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:12.034503 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" event={"ID":"246ac940-4fd5-40f9-b2e0-d341c1a90e6b","Type":"ContainerStarted","Data":"33c6b253bd19badc368204c96aedb7f9f6c8471431b8a2e0110e335a135d29f1"} Sep 05 20:11:12.034670 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:12.034565 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:12.056049 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:12.055992 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" podStartSLOduration=1.6325284500000001 podStartE2EDuration="2.055975547s" totalImagesPullingTime="423.447097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:11:12.054048102 +0000 UTC m=+1692.539348979" watchObservedRunningTime="2026-09-05 20:11:12.055975547 +0000 UTC m=+1692.541276419" Sep 05 20:11:18.045603 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:18.045570 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:11:38.982352 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:11:38.982274 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-s75vh_efeff32f-d46f-42d4-946f-a141c51b5ed4/payload-processing/0.log" Sep 05 20:12:00.318251 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:00.318218 2578 scope.go:117] "RemoveContainer" containerID="5b9d75f99679e85f86281229c5bb0b44ac814c34c9025dcd8018ba21013f4f7a" Sep 05 20:12:00.328465 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:00.328438 2578 scope.go:117] "RemoveContainer" containerID="cca9ca2b082e3d6cd80d09839b708407a5a1b1b92bdcf14f3036ab77abc0cc5d" Sep 05 20:12:14.778428 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:14.778395 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-s75vh_efeff32f-d46f-42d4-946f-a141c51b5ed4/payload-processing/0.log" Sep 05 20:12:20.374737 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.374700 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz"] Sep 05 20:12:20.378341 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.378325 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" Sep 05 20:12:20.380991 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.380964 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-14a23e\"" Sep 05 20:12:20.380991 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.380964 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-14a23e-dockercfg-ll74m\"" Sep 05 20:12:20.388846 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.388809 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz"] Sep 05 20:12:20.408812 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.408764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz8bl\" (UniqueName: \"kubernetes.io/projected/c6f18c8e-4feb-45c5-940f-d1723414a42a-kube-api-access-xz8bl\") pod \"payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz\" (UID: \"c6f18c8e-4feb-45c5-940f-d1723414a42a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" Sep 05 20:12:20.409005 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.408898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f18c8e-4feb-45c5-940f-d1723414a42a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz\" (UID: \"c6f18c8e-4feb-45c5-940f-d1723414a42a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" Sep 05 20:12:20.429465 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.429428 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf"] Sep 05 20:12:20.432892 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.432864 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" Sep 05 20:12:20.443283 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.443256 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf"] Sep 05 20:12:20.510058 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.510024 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c19d13f-c360-4577-b192-5ea8064691ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf\" (UID: \"0c19d13f-c360-4577-b192-5ea8064691ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" Sep 05 20:12:20.510224 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.510074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz8bl\" (UniqueName: \"kubernetes.io/projected/c6f18c8e-4feb-45c5-940f-d1723414a42a-kube-api-access-xz8bl\") pod \"payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz\" (UID: \"c6f18c8e-4feb-45c5-940f-d1723414a42a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" Sep 05 20:12:20.510224 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.510097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dshh9\" (UniqueName: \"kubernetes.io/projected/0c19d13f-c360-4577-b192-5ea8064691ac-kube-api-access-dshh9\") pod \"payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf\" (UID: \"0c19d13f-c360-4577-b192-5ea8064691ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" Sep 05 20:12:20.510224 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.510210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f18c8e-4feb-45c5-940f-d1723414a42a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz\" (UID: \"c6f18c8e-4feb-45c5-940f-d1723414a42a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" Sep 05 20:12:20.510890 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.510872 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f18c8e-4feb-45c5-940f-d1723414a42a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz\" (UID: \"c6f18c8e-4feb-45c5-940f-d1723414a42a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" Sep 05 20:12:20.525429 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.525399 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz8bl\" (UniqueName: \"kubernetes.io/projected/c6f18c8e-4feb-45c5-940f-d1723414a42a-kube-api-access-xz8bl\") pod \"payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz\" (UID: \"c6f18c8e-4feb-45c5-940f-d1723414a42a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" Sep 05 20:12:20.610697 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.610657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c19d13f-c360-4577-b192-5ea8064691ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf\" (UID: \"0c19d13f-c360-4577-b192-5ea8064691ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" Sep 05 20:12:20.610930 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.610717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dshh9\" (UniqueName: \"kubernetes.io/projected/0c19d13f-c360-4577-b192-5ea8064691ac-kube-api-access-dshh9\") pod \"payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf\" (UID: \"0c19d13f-c360-4577-b192-5ea8064691ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" Sep 05 20:12:20.611422 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.611399 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c19d13f-c360-4577-b192-5ea8064691ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf\" (UID: \"0c19d13f-c360-4577-b192-5ea8064691ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" Sep 05 20:12:20.619142 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.619116 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dshh9\" (UniqueName: \"kubernetes.io/projected/0c19d13f-c360-4577-b192-5ea8064691ac-kube-api-access-dshh9\") pod \"payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf\" (UID: \"0c19d13f-c360-4577-b192-5ea8064691ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" Sep 05 20:12:20.691121 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.691034 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" Sep 05 20:12:20.745143 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.745108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" Sep 05 20:12:20.965551 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:20.965525 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz"] Sep 05 20:12:20.968175 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:12:20.968146 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f18c8e_4feb_45c5_940f_d1723414a42a.slice/crio-1ff717e73db44bde1bfaf8dfdce9fae7c3534f5cb8c3f4a4e7c683856e25b39b WatchSource:0}: Error finding container 1ff717e73db44bde1bfaf8dfdce9fae7c3534f5cb8c3f4a4e7c683856e25b39b: Status 404 returned error can't find the container with id 1ff717e73db44bde1bfaf8dfdce9fae7c3534f5cb8c3f4a4e7c683856e25b39b Sep 05 20:12:21.035497 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:21.035471 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf"] Sep 05 20:12:21.037279 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:12:21.037257 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c19d13f_c360_4577_b192_5ea8064691ac.slice/crio-df2db47b41b854ac00df895089d3c54f85cee0f896808b91cc2af545fae87082 WatchSource:0}: Error finding container df2db47b41b854ac00df895089d3c54f85cee0f896808b91cc2af545fae87082: Status 404 returned error can't find the container with id df2db47b41b854ac00df895089d3c54f85cee0f896808b91cc2af545fae87082 Sep 05 20:12:21.378628 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:21.378591 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" event={"ID":"c6f18c8e-4feb-45c5-940f-d1723414a42a","Type":"ContainerStarted","Data":"e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201"} Sep 05 20:12:21.378628 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:21.378646 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" event={"ID":"c6f18c8e-4feb-45c5-940f-d1723414a42a","Type":"ContainerStarted","Data":"1ff717e73db44bde1bfaf8dfdce9fae7c3534f5cb8c3f4a4e7c683856e25b39b"} Sep 05 20:12:21.379170 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:21.378674 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" Sep 05 20:12:21.380329 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:21.380299 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" event={"ID":"0c19d13f-c360-4577-b192-5ea8064691ac","Type":"ContainerStarted","Data":"ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69"} Sep 05 20:12:21.380462 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:21.380349 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" event={"ID":"0c19d13f-c360-4577-b192-5ea8064691ac","Type":"ContainerStarted","Data":"df2db47b41b854ac00df895089d3c54f85cee0f896808b91cc2af545fae87082"} Sep 05 20:12:21.380462 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:21.380373 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" Sep 05 20:12:21.397689 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:21.397633 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" podStartSLOduration=1.397617201 podStartE2EDuration="1.397617201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:12:21.396626951 +0000 UTC m=+1761.881927825" watchObservedRunningTime="2026-09-05 20:12:21.397617201 +0000 UTC m=+1761.882918073" Sep 05 20:12:21.415230 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:21.415175 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" podStartSLOduration=1.415159671 podStartE2EDuration="1.415159671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:12:21.414618933 +0000 UTC m=+1761.899919807" watchObservedRunningTime="2026-09-05 20:12:21.415159671 +0000 UTC m=+1761.900460544" Sep 05 20:12:32.387689 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:32.387644 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" Sep 05 20:12:32.388174 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:32.387725 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" Sep 05 20:12:50.143732 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.143688 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf"] Sep 05 20:12:50.144214 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.143959 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" podUID="0c19d13f-c360-4577-b192-5ea8064691ac" containerName="payload-processing" containerID="cri-o://ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69" gracePeriod=30 Sep 05 20:12:50.148818 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.148763 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz"] Sep 05 20:12:50.149097 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.149037 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" podUID="c6f18c8e-4feb-45c5-940f-d1723414a42a" containerName="payload-pre-processing" containerID="cri-o://e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201" gracePeriod=30 Sep 05 20:12:50.414199 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.414175 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" Sep 05 20:12:50.443181 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.443153 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" Sep 05 20:12:50.511828 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.511765 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dshh9\" (UniqueName: \"kubernetes.io/projected/0c19d13f-c360-4577-b192-5ea8064691ac-kube-api-access-dshh9\") pod \"0c19d13f-c360-4577-b192-5ea8064691ac\" (UID: \"0c19d13f-c360-4577-b192-5ea8064691ac\") " Sep 05 20:12:50.512009 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.511905 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c19d13f-c360-4577-b192-5ea8064691ac-plugins-config-volume\") pod \"0c19d13f-c360-4577-b192-5ea8064691ac\" (UID: \"0c19d13f-c360-4577-b192-5ea8064691ac\") " Sep 05 20:12:50.512305 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.512280 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c19d13f-c360-4577-b192-5ea8064691ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c19d13f-c360-4577-b192-5ea8064691ac" (UID: "0c19d13f-c360-4577-b192-5ea8064691ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:12:50.514054 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.514028 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c19d13f-c360-4577-b192-5ea8064691ac-kube-api-access-dshh9" (OuterVolumeSpecName: "kube-api-access-dshh9") pod "0c19d13f-c360-4577-b192-5ea8064691ac" (UID: "0c19d13f-c360-4577-b192-5ea8064691ac"). InnerVolumeSpecName "kube-api-access-dshh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:12:50.528044 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.528006 2578 generic.go:356] "Generic (PLEG): container finished" podID="c6f18c8e-4feb-45c5-940f-d1723414a42a" containerID="e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201" exitCode=0 Sep 05 20:12:50.528176 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.528053 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" Sep 05 20:12:50.528176 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.528086 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" event={"ID":"c6f18c8e-4feb-45c5-940f-d1723414a42a","Type":"ContainerDied","Data":"e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201"} Sep 05 20:12:50.528176 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.528140 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz" event={"ID":"c6f18c8e-4feb-45c5-940f-d1723414a42a","Type":"ContainerDied","Data":"1ff717e73db44bde1bfaf8dfdce9fae7c3534f5cb8c3f4a4e7c683856e25b39b"} Sep 05 20:12:50.528176 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.528167 2578 scope.go:117] "RemoveContainer" containerID="e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201" Sep 05 20:12:50.529313 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.529296 2578 generic.go:356] "Generic (PLEG): container finished" podID="0c19d13f-c360-4577-b192-5ea8064691ac" containerID="ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69" exitCode=0 Sep 05 20:12:50.529380 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.529331 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" Sep 05 20:12:50.529421 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.529379 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" event={"ID":"0c19d13f-c360-4577-b192-5ea8064691ac","Type":"ContainerDied","Data":"ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69"} Sep 05 20:12:50.529467 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.529429 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf" event={"ID":"0c19d13f-c360-4577-b192-5ea8064691ac","Type":"ContainerDied","Data":"df2db47b41b854ac00df895089d3c54f85cee0f896808b91cc2af545fae87082"} Sep 05 20:12:50.536764 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.536747 2578 scope.go:117] "RemoveContainer" containerID="e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201" Sep 05 20:12:50.537107 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:12:50.537085 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201\": container with ID starting with e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201 not found: ID does not exist" containerID="e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201" Sep 05 20:12:50.537224 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.537112 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201"} err="failed to get container status \"e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201\": rpc error: code = NotFound desc = could not find container \"e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201\": container with ID starting with e8762317b3b2d83a202691153a7a85667dc44158b446b28857a821e9a1762201 not found: ID does not exist" Sep 05 20:12:50.537224 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.537129 2578 scope.go:117] "RemoveContainer" containerID="ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69" Sep 05 20:12:50.545675 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.545654 2578 scope.go:117] "RemoveContainer" containerID="ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69" Sep 05 20:12:50.546003 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:12:50.545985 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69\": container with ID starting with ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69 not found: ID does not exist" containerID="ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69" Sep 05 20:12:50.546084 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.546016 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69"} err="failed to get container status \"ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69\": rpc error: code = NotFound desc = could not find container \"ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69\": container with ID starting with ca57cf8fecd601f85e0c361fb1a40385916897bef6fa4a68b20cb75d96820f69 not found: ID does not exist" Sep 05 20:12:50.552247 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.552224 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf"] Sep 05 20:12:50.558068 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.558047 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-14a23e-5687d45d9f-m55vf"] Sep 05 20:12:50.613227 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.613183 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz8bl\" (UniqueName: \"kubernetes.io/projected/c6f18c8e-4feb-45c5-940f-d1723414a42a-kube-api-access-xz8bl\") pod \"c6f18c8e-4feb-45c5-940f-d1723414a42a\" (UID: \"c6f18c8e-4feb-45c5-940f-d1723414a42a\") " Sep 05 20:12:50.613414 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.613390 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f18c8e-4feb-45c5-940f-d1723414a42a-plugins-config-volume\") pod \"c6f18c8e-4feb-45c5-940f-d1723414a42a\" (UID: \"c6f18c8e-4feb-45c5-940f-d1723414a42a\") " Sep 05 20:12:50.613761 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.613728 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c19d13f-c360-4577-b192-5ea8064691ac-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:12:50.613761 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.613749 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dshh9\" (UniqueName: \"kubernetes.io/projected/0c19d13f-c360-4577-b192-5ea8064691ac-kube-api-access-dshh9\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:12:50.613761 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.613754 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f18c8e-4feb-45c5-940f-d1723414a42a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6f18c8e-4feb-45c5-940f-d1723414a42a" (UID: "c6f18c8e-4feb-45c5-940f-d1723414a42a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:12:50.615389 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.615360 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f18c8e-4feb-45c5-940f-d1723414a42a-kube-api-access-xz8bl" (OuterVolumeSpecName: "kube-api-access-xz8bl") pod "c6f18c8e-4feb-45c5-940f-d1723414a42a" (UID: "c6f18c8e-4feb-45c5-940f-d1723414a42a"). InnerVolumeSpecName "kube-api-access-xz8bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:12:50.715005 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.714921 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz8bl\" (UniqueName: \"kubernetes.io/projected/c6f18c8e-4feb-45c5-940f-d1723414a42a-kube-api-access-xz8bl\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:12:50.715005 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.714950 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f18c8e-4feb-45c5-940f-d1723414a42a-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:12:50.855567 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.855532 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz"] Sep 05 20:12:50.860713 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:50.860680 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-14a23e-554dcb8fc8-nwbbz"] Sep 05 20:12:52.012697 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:52.012659 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c19d13f-c360-4577-b192-5ea8064691ac" path="/var/lib/kubelet/pods/0c19d13f-c360-4577-b192-5ea8064691ac/volumes" Sep 05 20:12:52.013184 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:12:52.013089 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6f18c8e-4feb-45c5-940f-d1723414a42a" path="/var/lib/kubelet/pods/c6f18c8e-4feb-45c5-940f-d1723414a42a/volumes" Sep 05 20:13:00.102571 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:00.102543 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 20:13:00.111092 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:00.111071 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 20:13:15.746633 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:15.746594 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk"] Sep 05 20:13:15.747226 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:15.746822 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" podUID="55b7ba4b-8f12-4548-91db-93daa7e60386" containerName="maas-api" containerID="cri-o://cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625" gracePeriod=30 Sep 05 20:13:15.993677 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:15.993646 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:13:16.056768 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.056728 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgxwv\" (UniqueName: \"kubernetes.io/projected/55b7ba4b-8f12-4548-91db-93daa7e60386-kube-api-access-cgxwv\") pod \"55b7ba4b-8f12-4548-91db-93daa7e60386\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " Sep 05 20:13:16.056984 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.056842 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b7ba4b-8f12-4548-91db-93daa7e60386-maas-api-tls\") pod \"55b7ba4b-8f12-4548-91db-93daa7e60386\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " Sep 05 20:13:16.056984 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.056877 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b7ba4b-8f12-4548-91db-93daa7e60386-metrics-certs\") pod \"55b7ba4b-8f12-4548-91db-93daa7e60386\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " Sep 05 20:13:16.056984 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.056931 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b7ba4b-8f12-4548-91db-93daa7e60386-openshift-service-ca\") pod \"55b7ba4b-8f12-4548-91db-93daa7e60386\" (UID: \"55b7ba4b-8f12-4548-91db-93daa7e60386\") " Sep 05 20:13:16.057400 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.057370 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b7ba4b-8f12-4548-91db-93daa7e60386-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55b7ba4b-8f12-4548-91db-93daa7e60386" (UID: "55b7ba4b-8f12-4548-91db-93daa7e60386"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:13:16.058884 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.058861 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b7ba4b-8f12-4548-91db-93daa7e60386-kube-api-access-cgxwv" (OuterVolumeSpecName: "kube-api-access-cgxwv") pod "55b7ba4b-8f12-4548-91db-93daa7e60386" (UID: "55b7ba4b-8f12-4548-91db-93daa7e60386"). InnerVolumeSpecName "kube-api-access-cgxwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:13:16.058983 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.058949 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b7ba4b-8f12-4548-91db-93daa7e60386-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55b7ba4b-8f12-4548-91db-93daa7e60386" (UID: "55b7ba4b-8f12-4548-91db-93daa7e60386"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:13:16.058983 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.058970 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b7ba4b-8f12-4548-91db-93daa7e60386-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "55b7ba4b-8f12-4548-91db-93daa7e60386" (UID: "55b7ba4b-8f12-4548-91db-93daa7e60386"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:13:16.157642 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.157611 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b7ba4b-8f12-4548-91db-93daa7e60386-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:13:16.157642 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.157637 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b7ba4b-8f12-4548-91db-93daa7e60386-openshift-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:13:16.157642 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.157647 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgxwv\" (UniqueName: \"kubernetes.io/projected/55b7ba4b-8f12-4548-91db-93daa7e60386-kube-api-access-cgxwv\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:13:16.157901 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.157658 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b7ba4b-8f12-4548-91db-93daa7e60386-maas-api-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:13:16.661578 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.661547 2578 generic.go:356] "Generic (PLEG): container finished" podID="55b7ba4b-8f12-4548-91db-93daa7e60386" containerID="cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625" exitCode=0 Sep 05 20:13:16.661748 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.661591 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" event={"ID":"55b7ba4b-8f12-4548-91db-93daa7e60386","Type":"ContainerDied","Data":"cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625"} Sep 05 20:13:16.661748 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.661597 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" Sep 05 20:13:16.661748 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.661633 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk" event={"ID":"55b7ba4b-8f12-4548-91db-93daa7e60386","Type":"ContainerDied","Data":"923ef6f63d2ec111da0cd247badcb681b60b7ad208be96e91c5832aadf73e43a"} Sep 05 20:13:16.661748 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.661655 2578 scope.go:117] "RemoveContainer" containerID="cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625" Sep 05 20:13:16.670989 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.670968 2578 scope.go:117] "RemoveContainer" containerID="cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625" Sep 05 20:13:16.671253 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:13:16.671231 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625\": container with ID starting with cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625 not found: ID does not exist" containerID="cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625" Sep 05 20:13:16.671318 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.671265 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625"} err="failed to get container status \"cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625\": rpc error: code = NotFound desc = could not find container \"cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625\": container with ID starting with cf06683d831645bea80fe95e2a9834dbafadaa8724568d4d9ee980d7c742b625 not found: ID does not exist" Sep 05 20:13:16.682609 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.682581 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk"] Sep 05 20:13:16.684288 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:16.684266 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0f6b5-84ddf9c565-4wgsk"] Sep 05 20:13:18.012777 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:13:18.012740 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55b7ba4b-8f12-4548-91db-93daa7e60386" path="/var/lib/kubelet/pods/55b7ba4b-8f12-4548-91db-93daa7e60386/volumes" Sep 05 20:14:15.654952 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:15.654906 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw"] Sep 05 20:14:15.655902 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:15.655737 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" podUID="246ac940-4fd5-40f9-b2e0-d341c1a90e6b" containerName="maas-api" containerID="cri-o://941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d" gracePeriod=30 Sep 05 20:14:15.905531 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:15.905455 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:14:15.956753 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:15.956721 2578 generic.go:356] "Generic (PLEG): container finished" podID="246ac940-4fd5-40f9-b2e0-d341c1a90e6b" containerID="941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d" exitCode=0 Sep 05 20:14:15.956934 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:15.956768 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" Sep 05 20:14:15.956934 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:15.956780 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" event={"ID":"246ac940-4fd5-40f9-b2e0-d341c1a90e6b","Type":"ContainerDied","Data":"941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d"} Sep 05 20:14:15.956934 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:15.956845 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw" event={"ID":"246ac940-4fd5-40f9-b2e0-d341c1a90e6b","Type":"ContainerDied","Data":"33c6b253bd19badc368204c96aedb7f9f6c8471431b8a2e0110e335a135d29f1"} Sep 05 20:14:15.956934 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:15.956866 2578 scope.go:117] "RemoveContainer" containerID="941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d" Sep 05 20:14:15.965591 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:15.965572 2578 scope.go:117] "RemoveContainer" containerID="941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d" Sep 05 20:14:15.965912 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:14:15.965892 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d\": container with ID starting with 941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d not found: ID does not exist" containerID="941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d" Sep 05 20:14:15.965963 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:15.965924 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d"} err="failed to get container status \"941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d\": rpc error: code = NotFound desc = could not find container \"941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d\": container with ID starting with 941a04ad89c16d16dcf57be0d38e2bf24baecec0787ec410240dac82bf6b430d not found: ID does not exist" Sep 05 20:14:16.016253 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.016214 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-maas-api-tls\") pod \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " Sep 05 20:14:16.016663 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.016636 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-metrics-certs\") pod \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " Sep 05 20:14:16.016819 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.016709 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-openshift-service-ca\") pod \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " Sep 05 20:14:16.016819 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.016755 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brfls\" (UniqueName: \"kubernetes.io/projected/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-kube-api-access-brfls\") pod \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\" (UID: \"246ac940-4fd5-40f9-b2e0-d341c1a90e6b\") " Sep 05 20:14:16.017275 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.017240 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "246ac940-4fd5-40f9-b2e0-d341c1a90e6b" (UID: "246ac940-4fd5-40f9-b2e0-d341c1a90e6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:14:16.017415 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.017387 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-openshift-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:14:16.018688 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.018666 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "246ac940-4fd5-40f9-b2e0-d341c1a90e6b" (UID: "246ac940-4fd5-40f9-b2e0-d341c1a90e6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:14:16.019409 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.019379 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "246ac940-4fd5-40f9-b2e0-d341c1a90e6b" (UID: "246ac940-4fd5-40f9-b2e0-d341c1a90e6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:14:16.019488 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.019456 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-kube-api-access-brfls" (OuterVolumeSpecName: "kube-api-access-brfls") pod "246ac940-4fd5-40f9-b2e0-d341c1a90e6b" (UID: "246ac940-4fd5-40f9-b2e0-d341c1a90e6b"). InnerVolumeSpecName "kube-api-access-brfls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:14:16.118902 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.118865 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfls\" (UniqueName: \"kubernetes.io/projected/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-kube-api-access-brfls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:14:16.118902 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.118898 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-maas-api-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:14:16.118902 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.118908 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246ac940-4fd5-40f9-b2e0-d341c1a90e6b-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:14:16.280428 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.280398 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw"] Sep 05 20:14:16.281978 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:16.281955 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6693ae-6cb6f9f577-nngtw"] Sep 05 20:14:18.012521 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:18.012485 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="246ac940-4fd5-40f9-b2e0-d341c1a90e6b" path="/var/lib/kubelet/pods/246ac940-4fd5-40f9-b2e0-d341c1a90e6b/volumes" Sep 05 20:14:33.798784 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:33.798706 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx"] Sep 05 20:14:33.799247 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:33.798962 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" podUID="2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab" containerName="maas-api" containerID="cri-o://4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63" gracePeriod=30 Sep 05 20:14:34.042913 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.042886 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:14:34.051853 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.051744 2578 generic.go:356] "Generic (PLEG): container finished" podID="2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab" containerID="4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63" exitCode=0 Sep 05 20:14:34.051853 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.051787 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" Sep 05 20:14:34.051853 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.051822 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" event={"ID":"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab","Type":"ContainerDied","Data":"4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63"} Sep 05 20:14:34.052078 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.051869 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx" event={"ID":"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab","Type":"ContainerDied","Data":"4d02e22598c5ad0afd7c0ec3975db169fdd7c2ae3f8a657e89e033eed83d9931"} Sep 05 20:14:34.052078 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.051893 2578 scope.go:117] "RemoveContainer" containerID="4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63" Sep 05 20:14:34.060596 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.060574 2578 scope.go:117] "RemoveContainer" containerID="4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63" Sep 05 20:14:34.060923 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:14:34.060901 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63\": container with ID starting with 4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63 not found: ID does not exist" containerID="4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63" Sep 05 20:14:34.061004 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.060937 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63"} err="failed to get container status \"4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63\": rpc error: code = NotFound desc = could not find container \"4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63\": container with ID starting with 4a690204d65ce3816b2dc674d8510bc31a30aec24608de415566368c90d4bd63 not found: ID does not exist" Sep 05 20:14:34.089003 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.088970 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-maas-api-tls\") pod \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " Sep 05 20:14:34.089224 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.089034 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-metrics-certs\") pod \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " Sep 05 20:14:34.089284 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.089233 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-openshift-service-ca\") pod \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " Sep 05 20:14:34.089344 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.089285 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb6jk\" (UniqueName: \"kubernetes.io/projected/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-kube-api-access-jb6jk\") pod \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\" (UID: \"2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab\") " Sep 05 20:14:34.089914 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.089598 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab" (UID: "2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:14:34.089914 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.089725 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-openshift-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:14:34.091372 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.091348 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab" (UID: "2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:14:34.091579 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.091564 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab" (UID: "2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:14:34.091727 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.091709 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-kube-api-access-jb6jk" (OuterVolumeSpecName: "kube-api-access-jb6jk") pod "2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab" (UID: "2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab"). InnerVolumeSpecName "kube-api-access-jb6jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:14:34.191170 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.191131 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb6jk\" (UniqueName: \"kubernetes.io/projected/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-kube-api-access-jb6jk\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:14:34.191170 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.191164 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-maas-api-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:14:34.191170 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.191173 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:14:34.375674 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.375639 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx"] Sep 05 20:14:34.378062 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:34.378035 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4d1885-d646546d9-cnxbx"] Sep 05 20:14:36.011851 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:36.011816 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab" path="/var/lib/kubelet/pods/2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab/volumes" Sep 05 20:14:51.710984 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:51.710939 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv"] Sep 05 20:14:51.711478 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:51.711255 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" podUID="dfd4067e-ad23-44b8-9f41-c27b73760a47" containerName="maas-api" containerID="cri-o://17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757" gracePeriod=30 Sep 05 20:14:51.956976 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:51.956951 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:14:52.060402 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.060361 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlbkb\" (UniqueName: \"kubernetes.io/projected/dfd4067e-ad23-44b8-9f41-c27b73760a47-kube-api-access-nlbkb\") pod \"dfd4067e-ad23-44b8-9f41-c27b73760a47\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " Sep 05 20:14:52.060580 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.060418 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-maas-api-tls\") pod \"dfd4067e-ad23-44b8-9f41-c27b73760a47\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " Sep 05 20:14:52.060580 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.060502 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4067e-ad23-44b8-9f41-c27b73760a47-openshift-service-ca\") pod \"dfd4067e-ad23-44b8-9f41-c27b73760a47\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " Sep 05 20:14:52.060580 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.060521 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-metrics-certs\") pod \"dfd4067e-ad23-44b8-9f41-c27b73760a47\" (UID: \"dfd4067e-ad23-44b8-9f41-c27b73760a47\") " Sep 05 20:14:52.060918 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.060892 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd4067e-ad23-44b8-9f41-c27b73760a47-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dfd4067e-ad23-44b8-9f41-c27b73760a47" (UID: "dfd4067e-ad23-44b8-9f41-c27b73760a47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:14:52.062695 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.062673 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dfd4067e-ad23-44b8-9f41-c27b73760a47" (UID: "dfd4067e-ad23-44b8-9f41-c27b73760a47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:14:52.062814 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.062715 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dfd4067e-ad23-44b8-9f41-c27b73760a47" (UID: "dfd4067e-ad23-44b8-9f41-c27b73760a47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:14:52.062814 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.062728 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd4067e-ad23-44b8-9f41-c27b73760a47-kube-api-access-nlbkb" (OuterVolumeSpecName: "kube-api-access-nlbkb") pod "dfd4067e-ad23-44b8-9f41-c27b73760a47" (UID: "dfd4067e-ad23-44b8-9f41-c27b73760a47"). InnerVolumeSpecName "kube-api-access-nlbkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:14:52.140267 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.140227 2578 generic.go:356] "Generic (PLEG): container finished" podID="dfd4067e-ad23-44b8-9f41-c27b73760a47" containerID="17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757" exitCode=0 Sep 05 20:14:52.140267 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.140272 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" Sep 05 20:14:52.140526 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.140322 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" event={"ID":"dfd4067e-ad23-44b8-9f41-c27b73760a47","Type":"ContainerDied","Data":"17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757"} Sep 05 20:14:52.140526 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.140387 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv" event={"ID":"dfd4067e-ad23-44b8-9f41-c27b73760a47","Type":"ContainerDied","Data":"a6aca551366ef424fbdcc8bfc4f1942ed4da738fd9d5f93667384d8cf3599d6d"} Sep 05 20:14:52.140526 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.140417 2578 scope.go:117] "RemoveContainer" containerID="17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757" Sep 05 20:14:52.151398 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.151377 2578 scope.go:117] "RemoveContainer" containerID="17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757" Sep 05 20:14:52.151966 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:14:52.151941 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757\": container with ID starting with 17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757 not found: ID does not exist" containerID="17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757" Sep 05 20:14:52.152065 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.151973 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757"} err="failed to get container status \"17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757\": rpc error: code = NotFound desc = could not find container \"17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757\": container with ID starting with 17391526f928964b768e8157bf494fb4880de7db20693fc4700b47120764a757 not found: ID does not exist" Sep 05 20:14:52.161656 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.161627 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4067e-ad23-44b8-9f41-c27b73760a47-openshift-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:14:52.161767 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.161660 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:14:52.161767 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.161675 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlbkb\" (UniqueName: \"kubernetes.io/projected/dfd4067e-ad23-44b8-9f41-c27b73760a47-kube-api-access-nlbkb\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:14:52.161767 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.161687 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd4067e-ad23-44b8-9f41-c27b73760a47-maas-api-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:14:52.163368 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.163348 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv"] Sep 05 20:14:52.167924 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:52.167902 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0cc31a-78cdb46695-9ztdv"] Sep 05 20:14:54.012307 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:14:54.012271 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfd4067e-ad23-44b8-9f41-c27b73760a47" path="/var/lib/kubelet/pods/dfd4067e-ad23-44b8-9f41-c27b73760a47/volumes" Sep 05 20:15:00.380950 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:00.380918 2578 scope.go:117] "RemoveContainer" containerID="5191027716cbd5b419d4da94e62c628e0324121371df7e1c3a314266e984c746" Sep 05 20:15:01.748444 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:01.748408 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-7l6k5"] Sep 05 20:15:01.748847 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:01.748622 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" podUID="b9221c89-4902-4f7f-ae35-fd159704e009" containerName="manager" containerID="cri-o://8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34" gracePeriod=30 Sep 05 20:15:02.003633 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.003613 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 20:15:02.050357 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.050323 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w68x8\" (UniqueName: \"kubernetes.io/projected/b9221c89-4902-4f7f-ae35-fd159704e009-kube-api-access-w68x8\") pod \"b9221c89-4902-4f7f-ae35-fd159704e009\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " Sep 05 20:15:02.052525 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.052490 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9221c89-4902-4f7f-ae35-fd159704e009-kube-api-access-w68x8" (OuterVolumeSpecName: "kube-api-access-w68x8") pod "b9221c89-4902-4f7f-ae35-fd159704e009" (UID: "b9221c89-4902-4f7f-ae35-fd159704e009"). InnerVolumeSpecName "kube-api-access-w68x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:15:02.151577 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.151540 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-webhook-cert\") pod \"b9221c89-4902-4f7f-ae35-fd159704e009\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " Sep 05 20:15:02.151577 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.151588 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-metrics-certs\") pod \"b9221c89-4902-4f7f-ae35-fd159704e009\" (UID: \"b9221c89-4902-4f7f-ae35-fd159704e009\") " Sep 05 20:15:02.151868 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.151753 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w68x8\" (UniqueName: \"kubernetes.io/projected/b9221c89-4902-4f7f-ae35-fd159704e009-kube-api-access-w68x8\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:15:02.153752 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.153715 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b9221c89-4902-4f7f-ae35-fd159704e009" (UID: "b9221c89-4902-4f7f-ae35-fd159704e009"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:15:02.153752 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.153736 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b9221c89-4902-4f7f-ae35-fd159704e009" (UID: "b9221c89-4902-4f7f-ae35-fd159704e009"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:15:02.189071 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.189037 2578 generic.go:356] "Generic (PLEG): container finished" podID="b9221c89-4902-4f7f-ae35-fd159704e009" containerID="8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34" exitCode=0 Sep 05 20:15:02.189230 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.189079 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" event={"ID":"b9221c89-4902-4f7f-ae35-fd159704e009","Type":"ContainerDied","Data":"8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34"} Sep 05 20:15:02.189230 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.189084 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" Sep 05 20:15:02.189230 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.189119 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7686cbc6c4-7l6k5" event={"ID":"b9221c89-4902-4f7f-ae35-fd159704e009","Type":"ContainerDied","Data":"aa64845621a09e14270800b73fad79f2121d68a5892208a64a8e367035a01e69"} Sep 05 20:15:02.189230 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.189140 2578 scope.go:117] "RemoveContainer" containerID="8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34" Sep 05 20:15:02.198780 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.198760 2578 scope.go:117] "RemoveContainer" containerID="8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34" Sep 05 20:15:02.199086 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:15:02.199064 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34\": container with ID starting with 8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34 not found: ID does not exist" containerID="8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34" Sep 05 20:15:02.199135 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.199094 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34"} err="failed to get container status \"8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34\": rpc error: code = NotFound desc = could not find container \"8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34\": container with ID starting with 8d4fd956fdc248d086f5610eff15e71895d702558e796806b220f766f21ebc34 not found: ID does not exist" Sep 05 20:15:02.213201 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.213172 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-7l6k5"] Sep 05 20:15:02.217226 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.217203 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-7l6k5"] Sep 05 20:15:02.252977 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.252941 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-webhook-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:15:02.252977 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:02.252970 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9221c89-4902-4f7f-ae35-fd159704e009-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:15:04.012578 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:15:04.012540 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9221c89-4902-4f7f-ae35-fd159704e009" path="/var/lib/kubelet/pods/b9221c89-4902-4f7f-ae35-fd159704e009/volumes" Sep 05 20:16:02.995818 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.995760 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-f7r7m"] Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996113 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b7ba4b-8f12-4548-91db-93daa7e60386" containerName="maas-api" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996124 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b7ba4b-8f12-4548-91db-93daa7e60386" containerName="maas-api" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996134 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f18c8e-4feb-45c5-940f-d1723414a42a" containerName="payload-pre-processing" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996140 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f18c8e-4feb-45c5-940f-d1723414a42a" containerName="payload-pre-processing" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996150 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="246ac940-4fd5-40f9-b2e0-d341c1a90e6b" containerName="maas-api" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996156 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="246ac940-4fd5-40f9-b2e0-d341c1a90e6b" containerName="maas-api" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996167 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab" containerName="maas-api" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996172 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab" containerName="maas-api" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996183 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c19d13f-c360-4577-b192-5ea8064691ac" containerName="payload-processing" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996188 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c19d13f-c360-4577-b192-5ea8064691ac" containerName="payload-processing" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996196 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9221c89-4902-4f7f-ae35-fd159704e009" containerName="manager" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996201 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9221c89-4902-4f7f-ae35-fd159704e009" containerName="manager" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996207 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd4067e-ad23-44b8-9f41-c27b73760a47" containerName="maas-api" Sep 05 20:16:02.996220 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996212 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd4067e-ad23-44b8-9f41-c27b73760a47" containerName="maas-api" Sep 05 20:16:02.996658 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996267 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f18c8e-4feb-45c5-940f-d1723414a42a" containerName="payload-pre-processing" Sep 05 20:16:02.996658 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996276 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1d2e7d-0ae8-45c2-b290-b1b32bf4d3ab" containerName="maas-api" Sep 05 20:16:02.996658 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996282 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="55b7ba4b-8f12-4548-91db-93daa7e60386" containerName="maas-api" Sep 05 20:16:02.996658 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996288 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c19d13f-c360-4577-b192-5ea8064691ac" containerName="payload-processing" Sep 05 20:16:02.996658 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996293 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfd4067e-ad23-44b8-9f41-c27b73760a47" containerName="maas-api" Sep 05 20:16:02.996658 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996300 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9221c89-4902-4f7f-ae35-fd159704e009" containerName="manager" Sep 05 20:16:02.996658 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:02.996307 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="246ac940-4fd5-40f9-b2e0-d341c1a90e6b" containerName="maas-api" Sep 05 20:16:03.000344 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.000325 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:03.004502 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.004474 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 05 20:16:03.004645 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.004507 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 05 20:16:03.004645 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.004607 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d4vmp\"" Sep 05 20:16:03.004739 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.004635 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 05 20:16:03.012735 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.012710 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-f7r7m"] Sep 05 20:16:03.107501 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.107458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5vm4\" (UniqueName: \"kubernetes.io/projected/b6fdef14-461b-4d6d-bd87-74a77df6d694-kube-api-access-h5vm4\") pod \"maas-controller-7686cbc6c4-f7r7m\" (UID: \"b6fdef14-461b-4d6d-bd87-74a77df6d694\") " pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:03.107704 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.107525 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6fdef14-461b-4d6d-bd87-74a77df6d694-metrics-certs\") pod \"maas-controller-7686cbc6c4-f7r7m\" (UID: \"b6fdef14-461b-4d6d-bd87-74a77df6d694\") " pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:03.107704 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.107642 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6fdef14-461b-4d6d-bd87-74a77df6d694-webhook-cert\") pod \"maas-controller-7686cbc6c4-f7r7m\" (UID: \"b6fdef14-461b-4d6d-bd87-74a77df6d694\") " pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:03.208687 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.208643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6fdef14-461b-4d6d-bd87-74a77df6d694-webhook-cert\") pod \"maas-controller-7686cbc6c4-f7r7m\" (UID: \"b6fdef14-461b-4d6d-bd87-74a77df6d694\") " pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:03.208907 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.208699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vm4\" (UniqueName: \"kubernetes.io/projected/b6fdef14-461b-4d6d-bd87-74a77df6d694-kube-api-access-h5vm4\") pod \"maas-controller-7686cbc6c4-f7r7m\" (UID: \"b6fdef14-461b-4d6d-bd87-74a77df6d694\") " pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:03.208907 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.208740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6fdef14-461b-4d6d-bd87-74a77df6d694-metrics-certs\") pod \"maas-controller-7686cbc6c4-f7r7m\" (UID: \"b6fdef14-461b-4d6d-bd87-74a77df6d694\") " pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:03.211314 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.211291 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6fdef14-461b-4d6d-bd87-74a77df6d694-metrics-certs\") pod \"maas-controller-7686cbc6c4-f7r7m\" (UID: \"b6fdef14-461b-4d6d-bd87-74a77df6d694\") " pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:03.211411 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.211390 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6fdef14-461b-4d6d-bd87-74a77df6d694-webhook-cert\") pod \"maas-controller-7686cbc6c4-f7r7m\" (UID: \"b6fdef14-461b-4d6d-bd87-74a77df6d694\") " pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:03.218040 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.218015 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vm4\" (UniqueName: \"kubernetes.io/projected/b6fdef14-461b-4d6d-bd87-74a77df6d694-kube-api-access-h5vm4\") pod \"maas-controller-7686cbc6c4-f7r7m\" (UID: \"b6fdef14-461b-4d6d-bd87-74a77df6d694\") " pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:03.310006 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.309967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:03.582340 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.582312 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-f7r7m"] Sep 05 20:16:03.584672 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:16:03.584645 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fdef14_461b_4d6d_bd87_74a77df6d694.slice/crio-85f8ae75e982b9b166b7ced0e344e7add0951835940dd8ccd8fd66dd9d04c282 WatchSource:0}: Error finding container 85f8ae75e982b9b166b7ced0e344e7add0951835940dd8ccd8fd66dd9d04c282: Status 404 returned error can't find the container with id 85f8ae75e982b9b166b7ced0e344e7add0951835940dd8ccd8fd66dd9d04c282 Sep 05 20:16:03.586364 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:03.586347 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 20:16:04.480378 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:04.480281 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" event={"ID":"b6fdef14-461b-4d6d-bd87-74a77df6d694","Type":"ContainerStarted","Data":"88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc"} Sep 05 20:16:04.480378 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:04.480344 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" event={"ID":"b6fdef14-461b-4d6d-bd87-74a77df6d694","Type":"ContainerStarted","Data":"85f8ae75e982b9b166b7ced0e344e7add0951835940dd8ccd8fd66dd9d04c282"} Sep 05 20:16:04.480765 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:04.480407 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:04.500495 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:04.500445 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" podStartSLOduration=2.054246845 podStartE2EDuration="2.500425345s" totalImagesPullingTime="446.1785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:16:04.498317464 +0000 UTC m=+1984.983618362" watchObservedRunningTime="2026-09-05 20:16:04.500425345 +0000 UTC m=+1984.985726230" Sep 05 20:16:15.488187 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:15.488152 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:24.817746 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:24.817698 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-f7r7m"] Sep 05 20:16:24.818393 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:24.818000 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" podUID="b6fdef14-461b-4d6d-bd87-74a77df6d694" containerName="manager" containerID="cri-o://88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc" gracePeriod=30 Sep 05 20:16:25.062783 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.062754 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:25.114595 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.114510 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6fdef14-461b-4d6d-bd87-74a77df6d694-webhook-cert\") pod \"b6fdef14-461b-4d6d-bd87-74a77df6d694\" (UID: \"b6fdef14-461b-4d6d-bd87-74a77df6d694\") " Sep 05 20:16:25.114595 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.114583 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6fdef14-461b-4d6d-bd87-74a77df6d694-metrics-certs\") pod \"b6fdef14-461b-4d6d-bd87-74a77df6d694\" (UID: \"b6fdef14-461b-4d6d-bd87-74a77df6d694\") " Sep 05 20:16:25.114826 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.114650 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5vm4\" (UniqueName: \"kubernetes.io/projected/b6fdef14-461b-4d6d-bd87-74a77df6d694-kube-api-access-h5vm4\") pod \"b6fdef14-461b-4d6d-bd87-74a77df6d694\" (UID: \"b6fdef14-461b-4d6d-bd87-74a77df6d694\") " Sep 05 20:16:25.116913 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.116871 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6fdef14-461b-4d6d-bd87-74a77df6d694-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b6fdef14-461b-4d6d-bd87-74a77df6d694" (UID: "b6fdef14-461b-4d6d-bd87-74a77df6d694"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:16:25.116913 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.116897 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6fdef14-461b-4d6d-bd87-74a77df6d694-kube-api-access-h5vm4" (OuterVolumeSpecName: "kube-api-access-h5vm4") pod "b6fdef14-461b-4d6d-bd87-74a77df6d694" (UID: "b6fdef14-461b-4d6d-bd87-74a77df6d694"). InnerVolumeSpecName "kube-api-access-h5vm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:16:25.117143 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.117005 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6fdef14-461b-4d6d-bd87-74a77df6d694-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b6fdef14-461b-4d6d-bd87-74a77df6d694" (UID: "b6fdef14-461b-4d6d-bd87-74a77df6d694"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:16:25.215810 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.215764 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6fdef14-461b-4d6d-bd87-74a77df6d694-webhook-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:16:25.215984 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.215832 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6fdef14-461b-4d6d-bd87-74a77df6d694-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:16:25.215984 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.215844 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5vm4\" (UniqueName: \"kubernetes.io/projected/b6fdef14-461b-4d6d-bd87-74a77df6d694-kube-api-access-h5vm4\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:16:25.583786 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.583753 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6fdef14-461b-4d6d-bd87-74a77df6d694" containerID="88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc" exitCode=0 Sep 05 20:16:25.583974 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.583818 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" Sep 05 20:16:25.583974 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.583845 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" event={"ID":"b6fdef14-461b-4d6d-bd87-74a77df6d694","Type":"ContainerDied","Data":"88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc"} Sep 05 20:16:25.583974 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.583908 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7686cbc6c4-f7r7m" event={"ID":"b6fdef14-461b-4d6d-bd87-74a77df6d694","Type":"ContainerDied","Data":"85f8ae75e982b9b166b7ced0e344e7add0951835940dd8ccd8fd66dd9d04c282"} Sep 05 20:16:25.583974 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.583939 2578 scope.go:117] "RemoveContainer" containerID="88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc" Sep 05 20:16:25.593520 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.593501 2578 scope.go:117] "RemoveContainer" containerID="88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc" Sep 05 20:16:25.593843 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:16:25.593787 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc\": container with ID starting with 88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc not found: ID does not exist" containerID="88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc" Sep 05 20:16:25.593892 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.593852 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc"} err="failed to get container status \"88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc\": rpc error: code = NotFound desc = could not find container \"88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc\": container with ID starting with 88bd42081d2260f781045fba7b3bc619512987f9adeac9f7a5c9c31c1f5cb4cc not found: ID does not exist" Sep 05 20:16:25.607620 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.607595 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-f7r7m"] Sep 05 20:16:25.613333 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:25.613309 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-f7r7m"] Sep 05 20:16:26.012476 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.012443 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6fdef14-461b-4d6d-bd87-74a77df6d694" path="/var/lib/kubelet/pods/b6fdef14-461b-4d6d-bd87-74a77df6d694/volumes" Sep 05 20:16:26.385258 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.385176 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-rdbj2"] Sep 05 20:16:26.385534 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.385522 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6fdef14-461b-4d6d-bd87-74a77df6d694" containerName="manager" Sep 05 20:16:26.385576 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.385535 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6fdef14-461b-4d6d-bd87-74a77df6d694" containerName="manager" Sep 05 20:16:26.385623 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.385613 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6fdef14-461b-4d6d-bd87-74a77df6d694" containerName="manager" Sep 05 20:16:26.389941 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.389921 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:16:26.396913 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.396890 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 05 20:16:26.397049 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.396892 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d4vmp\"" Sep 05 20:16:26.397049 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.396894 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 05 20:16:26.397049 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.396897 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 05 20:16:26.406226 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.406198 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-rdbj2"] Sep 05 20:16:26.424140 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.424110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a-metrics-certs\") pod \"maas-controller-7686cbc6c4-rdbj2\" (UID: \"abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a\") " pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:16:26.424296 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.424157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a-webhook-cert\") pod \"maas-controller-7686cbc6c4-rdbj2\" (UID: \"abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a\") " pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:16:26.424296 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.424181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n865x\" (UniqueName: \"kubernetes.io/projected/abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a-kube-api-access-n865x\") pod \"maas-controller-7686cbc6c4-rdbj2\" (UID: \"abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a\") " pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:16:26.525552 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.525501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a-metrics-certs\") pod \"maas-controller-7686cbc6c4-rdbj2\" (UID: \"abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a\") " pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:16:26.525750 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.525583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a-webhook-cert\") pod \"maas-controller-7686cbc6c4-rdbj2\" (UID: \"abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a\") " pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:16:26.525750 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.525613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n865x\" (UniqueName: \"kubernetes.io/projected/abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a-kube-api-access-n865x\") pod \"maas-controller-7686cbc6c4-rdbj2\" (UID: \"abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a\") " pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:16:26.528143 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.528110 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a-metrics-certs\") pod \"maas-controller-7686cbc6c4-rdbj2\" (UID: \"abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a\") " pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:16:26.528256 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.528110 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a-webhook-cert\") pod \"maas-controller-7686cbc6c4-rdbj2\" (UID: \"abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a\") " pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:16:26.536992 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.536965 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n865x\" (UniqueName: \"kubernetes.io/projected/abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a-kube-api-access-n865x\") pod \"maas-controller-7686cbc6c4-rdbj2\" (UID: \"abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a\") " pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:16:26.699785 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.699692 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:16:26.976721 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:26.976679 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7686cbc6c4-rdbj2"] Sep 05 20:16:26.978591 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:16:26.978565 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcba8ee_5f2f_4bd0_8ba6_d1187fbbc11a.slice/crio-a5a8985a3b9c687d8f2a3a0f618dc99e78d4349b6c1a4ff5c9bcd78439c49b3a WatchSource:0}: Error finding container a5a8985a3b9c687d8f2a3a0f618dc99e78d4349b6c1a4ff5c9bcd78439c49b3a: Status 404 returned error can't find the container with id a5a8985a3b9c687d8f2a3a0f618dc99e78d4349b6c1a4ff5c9bcd78439c49b3a Sep 05 20:16:27.596613 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:27.596518 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" event={"ID":"abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a","Type":"ContainerStarted","Data":"b7cc1781c46875324e1a3d0c00fb6c1f1cf74c3198bcc33137c942438c40d23a"} Sep 05 20:16:27.596613 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:27.596581 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" event={"ID":"abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a","Type":"ContainerStarted","Data":"a5a8985a3b9c687d8f2a3a0f618dc99e78d4349b6c1a4ff5c9bcd78439c49b3a"} Sep 05 20:16:27.597053 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:27.596745 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:16:27.615933 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:27.615874 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" podStartSLOduration=1.294583531 podStartE2EDuration="1.61585862s" totalImagesPullingTime="321.275089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:16:27.614619992 +0000 UTC m=+2008.099920866" watchObservedRunningTime="2026-09-05 20:16:27.61585862 +0000 UTC m=+2008.101159493" Sep 05 20:16:38.603655 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:16:38.603622 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7686cbc6c4-rdbj2" Sep 05 20:18:00.130257 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:18:00.130229 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 20:18:00.139975 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:18:00.139952 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kn472_bfd09d5d-9d6d-4b49-b871-10b71cd7eff2/ovn-acl-logging/0.log" Sep 05 20:20:18.373904 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.373867 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6886844f88-mvdr9"] Sep 05 20:20:18.377226 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.377208 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.387206 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.387172 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6886844f88-mvdr9"] Sep 05 20:20:18.472533 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.472485 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fb8a72-af4a-4583-a6e1-4e39819632ad-metrics-certs\") pod \"maas-api-6886844f88-mvdr9\" (UID: \"d4fb8a72-af4a-4583-a6e1-4e39819632ad\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.472756 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.472567 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fb8a72-af4a-4583-a6e1-4e39819632ad-openshift-service-ca\") pod \"maas-api-6886844f88-mvdr9\" (UID: \"d4fb8a72-af4a-4583-a6e1-4e39819632ad\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.472756 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.472638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w9nb\" (UniqueName: \"kubernetes.io/projected/d4fb8a72-af4a-4583-a6e1-4e39819632ad-kube-api-access-8w9nb\") pod \"maas-api-6886844f88-mvdr9\" (UID: \"d4fb8a72-af4a-4583-a6e1-4e39819632ad\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.472756 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.472683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fb8a72-af4a-4583-a6e1-4e39819632ad-maas-api-tls\") pod \"maas-api-6886844f88-mvdr9\" (UID: \"d4fb8a72-af4a-4583-a6e1-4e39819632ad\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.573363 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.573319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w9nb\" (UniqueName: \"kubernetes.io/projected/d4fb8a72-af4a-4583-a6e1-4e39819632ad-kube-api-access-8w9nb\") pod \"maas-api-6886844f88-mvdr9\" (UID: \"d4fb8a72-af4a-4583-a6e1-4e39819632ad\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.573561 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.573384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fb8a72-af4a-4583-a6e1-4e39819632ad-maas-api-tls\") pod \"maas-api-6886844f88-mvdr9\" (UID: \"d4fb8a72-af4a-4583-a6e1-4e39819632ad\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.573561 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.573412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fb8a72-af4a-4583-a6e1-4e39819632ad-metrics-certs\") pod \"maas-api-6886844f88-mvdr9\" (UID: \"d4fb8a72-af4a-4583-a6e1-4e39819632ad\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.573561 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.573457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fb8a72-af4a-4583-a6e1-4e39819632ad-openshift-service-ca\") pod \"maas-api-6886844f88-mvdr9\" (UID: \"d4fb8a72-af4a-4583-a6e1-4e39819632ad\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.574134 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.574109 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fb8a72-af4a-4583-a6e1-4e39819632ad-openshift-service-ca\") pod \"maas-api-6886844f88-mvdr9\" (UID: \"d4fb8a72-af4a-4583-a6e1-4e39819632ad\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.575989 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.575967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fb8a72-af4a-4583-a6e1-4e39819632ad-metrics-certs\") pod \"maas-api-6886844f88-mvdr9\" (UID: \"d4fb8a72-af4a-4583-a6e1-4e39819632ad\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.576123 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.576017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fb8a72-af4a-4583-a6e1-4e39819632ad-maas-api-tls\") pod \"maas-api-6886844f88-mvdr9\" (UID: \"d4fb8a72-af4a-4583-a6e1-4e39819632ad\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.580684 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.580658 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w9nb\" (UniqueName: \"kubernetes.io/projected/d4fb8a72-af4a-4583-a6e1-4e39819632ad-kube-api-access-8w9nb\") pod \"maas-api-6886844f88-mvdr9\" (UID: \"d4fb8a72-af4a-4583-a6e1-4e39819632ad\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.689444 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.689351 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:18.966654 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:18.966446 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6886844f88-mvdr9"] Sep 05 20:20:18.969648 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:20:18.969614 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fb8a72_af4a_4583_a6e1_4e39819632ad.slice/crio-2492da0bb59c5ca98a1a1a128dc33d44d1922ea28553c7ea4fe9500de324bc08 WatchSource:0}: Error finding container 2492da0bb59c5ca98a1a1a128dc33d44d1922ea28553c7ea4fe9500de324bc08: Status 404 returned error can't find the container with id 2492da0bb59c5ca98a1a1a128dc33d44d1922ea28553c7ea4fe9500de324bc08 Sep 05 20:20:19.689063 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:19.688970 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" event={"ID":"d4fb8a72-af4a-4583-a6e1-4e39819632ad","Type":"ContainerStarted","Data":"4f028768c0df66c3f30cc53a55900920f44c6fec1df97e55c03d67e91e1826d6"} Sep 05 20:20:19.689063 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:19.689030 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" event={"ID":"d4fb8a72-af4a-4583-a6e1-4e39819632ad","Type":"ContainerStarted","Data":"2492da0bb59c5ca98a1a1a128dc33d44d1922ea28553c7ea4fe9500de324bc08"} Sep 05 20:20:19.689455 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:19.689074 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:19.705517 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:19.705467 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" podStartSLOduration=1.344376819 podStartE2EDuration="1.705450175s" totalImagesPullingTime="361.073356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:20:19.704261417 +0000 UTC m=+2240.189562292" watchObservedRunningTime="2026-09-05 20:20:19.705450175 +0000 UTC m=+2240.190751048" Sep 05 20:20:25.698310 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:25.698229 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6886844f88-mvdr9" Sep 05 20:20:33.029292 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:33.029257 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr"] Sep 05 20:20:33.029688 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:33.029501 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" podUID="16a6cb41-08fa-45f0-9ebf-89b5c8c9b996" containerName="maas-api" containerID="cri-o://92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa" gracePeriod=30 Sep 05 20:20:33.143933 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:33.143886 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" podUID="16a6cb41-08fa-45f0-9ebf-89b5c8c9b996" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.34:8443/health\": dial tcp 10.132.0.34:8443: connect: connection refused" Sep 05 20:20:34.277273 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.277244 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 20:20:34.416523 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.416433 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-metrics-certs\") pod \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " Sep 05 20:20:34.416523 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.416473 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-maas-api-tls\") pod \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " Sep 05 20:20:34.416735 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.416565 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-openshift-service-ca\") pod \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " Sep 05 20:20:34.416735 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.416636 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkhtr\" (UniqueName: \"kubernetes.io/projected/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-kube-api-access-bkhtr\") pod \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\" (UID: \"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996\") " Sep 05 20:20:34.416983 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.416953 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16a6cb41-08fa-45f0-9ebf-89b5c8c9b996" (UID: "16a6cb41-08fa-45f0-9ebf-89b5c8c9b996"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 20:20:34.418707 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.418677 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-kube-api-access-bkhtr" (OuterVolumeSpecName: "kube-api-access-bkhtr") pod "16a6cb41-08fa-45f0-9ebf-89b5c8c9b996" (UID: "16a6cb41-08fa-45f0-9ebf-89b5c8c9b996"). InnerVolumeSpecName "kube-api-access-bkhtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 20:20:34.418864 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.418739 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "16a6cb41-08fa-45f0-9ebf-89b5c8c9b996" (UID: "16a6cb41-08fa-45f0-9ebf-89b5c8c9b996"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:20:34.418864 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.418750 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16a6cb41-08fa-45f0-9ebf-89b5c8c9b996" (UID: "16a6cb41-08fa-45f0-9ebf-89b5c8c9b996"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 20:20:34.518039 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.518005 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-metrics-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:20:34.518039 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.518036 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-maas-api-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:20:34.518039 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.518045 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-openshift-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:20:34.518261 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.518055 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkhtr\" (UniqueName: \"kubernetes.io/projected/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996-kube-api-access-bkhtr\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\"" Sep 05 20:20:34.761764 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.761728 2578 generic.go:356] "Generic (PLEG): container finished" podID="16a6cb41-08fa-45f0-9ebf-89b5c8c9b996" containerID="92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa" exitCode=0 Sep 05 20:20:34.761949 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.761772 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" Sep 05 20:20:34.761949 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.761780 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" event={"ID":"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996","Type":"ContainerDied","Data":"92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa"} Sep 05 20:20:34.761949 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.761877 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr" event={"ID":"16a6cb41-08fa-45f0-9ebf-89b5c8c9b996","Type":"ContainerDied","Data":"917c904a03ab2b5b306a663d19f9d1d26c97d12d501362f9d206176d97baba0e"} Sep 05 20:20:34.761949 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.761903 2578 scope.go:117] "RemoveContainer" containerID="92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa" Sep 05 20:20:34.770971 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.770949 2578 scope.go:117] "RemoveContainer" containerID="92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa" Sep 05 20:20:34.771258 ip-10-0-143-252 kubenswrapper[2578]: E0905 20:20:34.771241 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa\": container with ID starting with 92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa not found: ID does not exist" containerID="92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa" Sep 05 20:20:34.771334 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.771264 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa"} err="failed to get container status \"92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa\": rpc error: code = NotFound desc = could not find container \"92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa\": container with ID starting with 92eecfdc330ba4a9656a7e5698bd898ecf4f111962a3509c731e983ef99e72aa not found: ID does not exist" Sep 05 20:20:34.782073 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.782045 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr"] Sep 05 20:20:34.788090 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:34.788062 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-n4qfr"] Sep 05 20:20:36.012169 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:36.012131 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16a6cb41-08fa-45f0-9ebf-89b5c8c9b996" path="/var/lib/kubelet/pods/16a6cb41-08fa-45f0-9ebf-89b5c8c9b996/volumes" Sep 05 20:20:53.008829 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:53.008314 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7686cbc6c4-rdbj2_abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a/manager/0.log" Sep 05 20:20:53.462180 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:53.462093 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-588fc9dcd8-kv67g_d49fb878-4312-4d3c-92d7-f0654de8071d/copy-manifests/0.log" Sep 05 20:20:53.468557 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:53.468523 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-588fc9dcd8-kv67g_d49fb878-4312-4d3c-92d7-f0654de8071d/manager/0.log" Sep 05 20:20:54.031620 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:54.031586 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6886844f88-mvdr9_d4fb8a72-af4a-4583-a6e1-4e39819632ad/maas-api/0.log" Sep 05 20:20:54.372143 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:54.372057 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-x9s9x_bbaffa32-4a72-4204-bb51-e4bd03a27113/postgres/0.log" Sep 05 20:20:56.413571 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:56.413535 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84fbbb6876-9dwpv_617ea30d-e653-4a6f-882b-3ea090eee8ca/kube-auth-proxy/0.log" Sep 05 20:20:56.751959 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:56.751872 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-s75vh_efeff32f-d46f-42d4-946f-a141c51b5ed4/payload-processing/0.log" Sep 05 20:20:57.284711 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:57.284677 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-xxqq2_09074706-f21e-40dc-b564-fe4930e934ad/main/0.log" Sep 05 20:20:57.500603 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:57.500564 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-kc5rv_19c6907e-797d-4cf7-9519-1434ec53b2ce/main/0.log" Sep 05 20:20:57.610901 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:57.610819 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnc27t_d2d3c008-dab1-4880-bbdc-c606698e9938/main/0.log" Sep 05 20:20:57.830881 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:20:57.830851 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6srkq_fc5c8ede-535a-459c-9a55-bbfb28fe51fd/main/0.log" Sep 05 20:21:06.222401 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:06.222368 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4l4q6_1ff9d244-8eb8-4b0d-ad2d-618e28803d5a/global-pull-secret-syncer/0.log" Sep 05 20:21:06.421025 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:06.420990 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wcqth_452e1429-08ff-4b09-98b0-2a5f69bffa1a/konnectivity-agent/0.log" Sep 05 20:21:06.491368 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:06.491266 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-252.ec2.internal_dfffd2d51175b09bb92577df4eb5ea66/haproxy/0.log" Sep 05 20:21:12.937956 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:12.937923 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7cbf6b79f5-cqqhr_42dd5b05-f773-4a2f-96ba-cf5217e3841d/metrics-server/0.log" Sep 05 20:21:12.970150 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:12.970124 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-fxzkk_df4f33b8-cae3-4624-b9ef-0c4cd2ba4951/monitoring-plugin/0.log" Sep 05 20:21:13.003743 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.003713 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-298md_a1254285-9e7d-4ad2-b3c7-5d446e29f0fe/node-exporter/0.log" Sep 05 20:21:13.024011 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.023984 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-298md_a1254285-9e7d-4ad2-b3c7-5d446e29f0fe/kube-rbac-proxy/0.log" Sep 05 20:21:13.043806 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.043768 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-298md_a1254285-9e7d-4ad2-b3c7-5d446e29f0fe/init-textfile/0.log" Sep 05 20:21:13.331324 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.331288 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c46619e-0f25-424d-9d30-b6e50d11bb14/prometheus/0.log" Sep 05 20:21:13.357149 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.357111 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c46619e-0f25-424d-9d30-b6e50d11bb14/config-reloader/0.log" Sep 05 20:21:13.378379 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.378346 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c46619e-0f25-424d-9d30-b6e50d11bb14/thanos-sidecar/0.log" Sep 05 20:21:13.397971 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.397941 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c46619e-0f25-424d-9d30-b6e50d11bb14/kube-rbac-proxy-web/0.log" Sep 05 20:21:13.420681 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.420648 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c46619e-0f25-424d-9d30-b6e50d11bb14/kube-rbac-proxy/0.log" Sep 05 20:21:13.440668 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.440644 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c46619e-0f25-424d-9d30-b6e50d11bb14/kube-rbac-proxy-thanos/0.log" Sep 05 20:21:13.462565 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.462537 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c46619e-0f25-424d-9d30-b6e50d11bb14/init-config-reloader/0.log" Sep 05 20:21:13.492273 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.492247 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-2f4j7_392a4d0a-31cb-43f7-9a22-695c9a54ad02/prometheus-operator/0.log" Sep 05 20:21:13.510021 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.509995 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-2f4j7_392a4d0a-31cb-43f7-9a22-695c9a54ad02/kube-rbac-proxy/0.log" Sep 05 20:21:13.536466 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:13.536441 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-fqlsc_1cf40c05-15d3-46fa-9aeb-5296fdd7b3c9/prometheus-operator-admission-webhook/0.log" Sep 05 20:21:15.346808 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:15.346776 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-rm5lj_f57d845b-1d2a-4c86-abf5-2f72a04b4532/networking-console-plugin/0.log" Sep 05 20:21:16.584414 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:16.584382 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-n5qrf_7ae64473-c8b8-42cf-bc67-c7d67f744d2c/download-server/0.log" Sep 05 20:21:17.196140 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.196101 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb"] Sep 05 20:21:17.196585 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.196552 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16a6cb41-08fa-45f0-9ebf-89b5c8c9b996" containerName="maas-api" Sep 05 20:21:17.196585 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.196576 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="16a6cb41-08fa-45f0-9ebf-89b5c8c9b996" containerName="maas-api" Sep 05 20:21:17.196845 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.196670 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="16a6cb41-08fa-45f0-9ebf-89b5c8c9b996" containerName="maas-api" Sep 05 20:21:17.199928 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.199907 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.202303 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.202282 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hdj6r\"/\"kube-root-ca.crt\"" Sep 05 20:21:17.202423 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.202402 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hdj6r\"/\"openshift-service-ca.crt\"" Sep 05 20:21:17.203503 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.203488 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hdj6r\"/\"default-dockercfg-bdppc\"" Sep 05 20:21:17.208991 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.208967 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb"] Sep 05 20:21:17.330777 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.330739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-sys\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.331010 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.330786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-proc\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.331010 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.330875 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-lib-modules\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.331010 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.330916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-podres\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.331010 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.330964 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6hv\" (UniqueName: \"kubernetes.io/projected/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-kube-api-access-qf6hv\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.432384 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.432342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-sys\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.432384 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.432391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-proc\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.432642 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.432421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-lib-modules\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.432642 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.432443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-podres\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.432642 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.432492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6hv\" (UniqueName: \"kubernetes.io/projected/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-kube-api-access-qf6hv\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.432642 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.432489 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-sys\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.432642 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.432544 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-proc\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.432642 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.432615 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-lib-modules\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.432642 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.432635 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-podres\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.440439 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.440414 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6hv\" (UniqueName: \"kubernetes.io/projected/964a0ec6-b29e-497a-86d8-748cc1dc6b1d-kube-api-access-qf6hv\") pod \"perf-node-gather-daemonset-24fhb\" (UID: \"964a0ec6-b29e-497a-86d8-748cc1dc6b1d\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.510108 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.510065 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:17.782961 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.782915 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb"] Sep 05 20:21:17.785211 ip-10-0-143-252 kubenswrapper[2578]: W0905 20:21:17.785176 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod964a0ec6_b29e_497a_86d8_748cc1dc6b1d.slice/crio-fe5048b9fe06892bd243cf7e691cf496fe927ae443e85006bc42b1b4094149b2 WatchSource:0}: Error finding container fe5048b9fe06892bd243cf7e691cf496fe927ae443e85006bc42b1b4094149b2: Status 404 returned error can't find the container with id fe5048b9fe06892bd243cf7e691cf496fe927ae443e85006bc42b1b4094149b2 Sep 05 20:21:17.972328 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.972292 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" event={"ID":"964a0ec6-b29e-497a-86d8-748cc1dc6b1d","Type":"ContainerStarted","Data":"7fadc873680b3936eb2b5efe4a7d25d57529b4e4dbd766cba04a6eb4be57dd46"} Sep 05 20:21:17.972508 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.972348 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" event={"ID":"964a0ec6-b29e-497a-86d8-748cc1dc6b1d","Type":"ContainerStarted","Data":"fe5048b9fe06892bd243cf7e691cf496fe927ae443e85006bc42b1b4094149b2"} Sep 05 20:21:17.972508 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:17.972374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:18.003690 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:18.003625 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" podStartSLOduration=1.003606948 podStartE2EDuration="1.003606948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 20:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:21:18.001646745 +0000 UTC m=+2298.486947618" watchObservedRunningTime="2026-09-05 20:21:18.003606948 +0000 UTC m=+2298.488907878" Sep 05 20:21:18.558893 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:18.558862 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rzzf9_99e4b881-669e-4df5-9feb-e59d740252bf/dns/0.log" Sep 05 20:21:18.578566 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:18.578538 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rzzf9_99e4b881-669e-4df5-9feb-e59d740252bf/kube-rbac-proxy/0.log" Sep 05 20:21:18.690067 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:18.690040 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-26ptb_593dff6f-9c82-4da6-90cb-35032860e715/dns-node-resolver/0.log" Sep 05 20:21:19.314389 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:19.314345 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qc2c8_b04d888b-736a-42e7-8a96-81d738b05abb/node-ca/0.log" Sep 05 20:21:20.803032 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:20.802996 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84fbbb6876-9dwpv_617ea30d-e653-4a6f-882b-3ea090eee8ca/kube-auth-proxy/0.log" Sep 05 20:21:20.929670 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:20.929630 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-s75vh_efeff32f-d46f-42d4-946f-a141c51b5ed4/payload-processing/0.log" Sep 05 20:21:21.855907 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:21.855877 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vkt5p_9e1257f4-7b80-40cb-b11d-eb17481f369e/serve-healthcheck-canary/0.log" Sep 05 20:21:22.317444 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:22.317403 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-nptkb_66922fa5-4d55-479f-b0b5-baef39b284f3/insights-operator/0.log" Sep 05 20:21:22.317628 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:22.317572 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-nptkb_66922fa5-4d55-479f-b0b5-baef39b284f3/insights-operator/1.log" Sep 05 20:21:22.336962 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:22.336933 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-85hn8_a9891d97-10d1-45d7-92c2-b2e28465c386/kube-rbac-proxy/0.log" Sep 05 20:21:22.357434 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:22.357410 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-85hn8_a9891d97-10d1-45d7-92c2-b2e28465c386/exporter/0.log" Sep 05 20:21:22.377235 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:22.377204 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-85hn8_a9891d97-10d1-45d7-92c2-b2e28465c386/extractor/0.log" Sep 05 20:21:23.986651 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:23.986622 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-24fhb" Sep 05 20:21:24.835329 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:24.835295 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7686cbc6c4-rdbj2_abcba8ee-5f2f-4bd0-8ba6-d1187fbbc11a/manager/0.log" Sep 05 20:21:25.069265 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:25.069233 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-588fc9dcd8-kv67g_d49fb878-4312-4d3c-92d7-f0654de8071d/manager/0.log" Sep 05 20:21:25.093943 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:25.093858 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-588fc9dcd8-kv67g_d49fb878-4312-4d3c-92d7-f0654de8071d/copy-manifests/0.log" Sep 05 20:21:26.662136 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:26.662108 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6b9f469976-tscz7_ae9ff93a-5782-4138-8d45-40ddc61ea16d/manager/0.log" Sep 05 20:21:26.697311 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:26.697225 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-8q5nb_286a579d-14bd-4fef-a378-8dbef9fb1d7d/openshift-lws-operator/0.log" Sep 05 20:21:28.002024 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:28.001993 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84fbbb6876-9dwpv_617ea30d-e653-4a6f-882b-3ea090eee8ca/kube-auth-proxy/0.log" Sep 05 20:21:28.113938 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:28.113891 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-s75vh_efeff32f-d46f-42d4-946f-a141c51b5ed4/payload-processing/0.log" Sep 05 20:21:30.302300 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:30.302274 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84fbbb6876-9dwpv_617ea30d-e653-4a6f-882b-3ea090eee8ca/kube-auth-proxy/0.log" Sep 05 20:21:30.400568 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:30.400534 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-s75vh_efeff32f-d46f-42d4-946f-a141c51b5ed4/payload-processing/0.log" Sep 05 20:21:33.463097 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:33.463066 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpglw_51a28ad1-9a9b-47be-91b2-92dddc4a1408/kube-multus-additional-cni-plugins/0.log" Sep 05 20:21:33.482841 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:33.482783 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpglw_51a28ad1-9a9b-47be-91b2-92dddc4a1408/egress-router-binary-copy/0.log" Sep 05 20:21:33.502002 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:33.501972 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpglw_51a28ad1-9a9b-47be-91b2-92dddc4a1408/cni-plugins/0.log" Sep 05 20:21:33.521421 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:33.521368 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpglw_51a28ad1-9a9b-47be-91b2-92dddc4a1408/bond-cni-plugin/0.log" Sep 05 20:21:33.541607 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:33.541566 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpglw_51a28ad1-9a9b-47be-91b2-92dddc4a1408/routeoverride-cni/0.log" Sep 05 20:21:33.561356 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:33.561312 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpglw_51a28ad1-9a9b-47be-91b2-92dddc4a1408/whereabouts-cni-bincopy/0.log" Sep 05 20:21:33.581631 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:33.581593 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpglw_51a28ad1-9a9b-47be-91b2-92dddc4a1408/whereabouts-cni/0.log" Sep 05 20:21:33.839363 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:33.839325 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-w7qgd_8901e999-5de1-4671-8d64-7f40616f1667/kube-multus/0.log" Sep 05 20:21:33.907557 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:33.907517 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-89gvd_fe2aaad4-dc65-4f15-be9a-9fe0babba5b7/network-metrics-daemon/0.log" Sep 05 20:21:33.923496 ip-10-0-143-252 kubenswrapper[2578]: I0905 20:21:33.923460 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-89gvd_fe2aaad4-dc65-4f15-be9a-9fe0babba5b7/kube-rbac-proxy/0.log"