Sep 06 05:53:53.043350 ip-10-0-142-34 systemd[1]: Starting Kubernetes Kubelet... Sep 06 05:53:53.470822 ip-10-0-142-34 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 05:53:53.470822 ip-10-0-142-34 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 06 05:53:53.470822 ip-10-0-142-34 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 05:53:53.470822 ip-10-0-142-34 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 06 05:53:53.471506 ip-10-0-142-34 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 05:53:53.472285 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.472196 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 06 05:53:53.477788 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477771 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 05:53:53.477788 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477787 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477791 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477794 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477797 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477799 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477802 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477805 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477808 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477811 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477814 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477817 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477820 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477822 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477825 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477828 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477830 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477833 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477837 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477839 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477842 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 05:53:53.477852 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477844 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477850 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477853 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477856 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477859 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477862 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477865 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477868 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477870 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477873 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477875 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477878 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477881 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477883 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477886 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477888 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477892 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477896 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477899 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 05:53:53.478345 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477901 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477905 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477909 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477914 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477916 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477919 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477922 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477926 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477929 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477932 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477935 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477937 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477940 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477942 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477945 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477948 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477951 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477953 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477957 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 05:53:53.478814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477960 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477963 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477965 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477968 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477970 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477974 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477976 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477979 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477981 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477984 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477987 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477989 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477992 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477994 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477997 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.477999 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478002 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478004 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478007 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478009 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 05:53:53.479327 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478012 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478015 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478018 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478020 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478023 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478025 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478027 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478031 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478037 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478052 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478056 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478060 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478063 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478066 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478068 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478071 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478074 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478077 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478080 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 05:53:53.479814 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478083 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478800 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478807 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478810 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478812 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478816 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478835 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478840 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478843 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478846 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478849 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478852 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478855 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478857 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478860 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478864 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478866 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478869 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478871 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478874 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 05:53:53.480290 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478877 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478880 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478883 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478886 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478888 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478891 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478893 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478896 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478898 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478901 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478903 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478906 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478908 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478911 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478914 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478916 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478919 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478921 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478924 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478926 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478929 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 05:53:53.480787 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478932 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478934 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478937 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478942 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478945 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478948 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478951 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478954 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478957 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478959 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478962 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478964 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478967 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478970 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478973 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478976 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478979 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478981 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478984 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 05:53:53.481339 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478987 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478989 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478992 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478995 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.478997 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479000 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479002 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479005 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479008 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479010 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479012 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479015 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479018 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479021 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479024 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479027 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479030 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479032 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479034 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 05:53:53.481843 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479037 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479039 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479055 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479058 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479061 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479064 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479067 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479070 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479072 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479075 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479077 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479081 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479083 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479086 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479088 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479093 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479096 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479100 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479102 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 06 05:53:53.482322 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479105 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479108 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479483 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479490 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479493 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479496 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479499 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479501 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479504 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479508 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479512 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479515 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479517 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479520 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479522 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479526 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479528 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479531 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479534 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479538 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 05:53:53.482794 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479540 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479543 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479545 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479548 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479550 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479553 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479555 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479558 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479560 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479563 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479565 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479568 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479571 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479573 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479576 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479580 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479582 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479585 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479587 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479590 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 05:53:53.483293 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479592 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479594 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479597 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479601 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479603 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479606 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479608 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479610 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479614 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479616 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479619 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479624 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479627 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479630 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479633 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479636 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479638 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479641 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479644 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479647 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 05:53:53.483786 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479649 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479652 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479655 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479657 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479660 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479662 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479665 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479668 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479671 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479673 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479676 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479680 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479682 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479685 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479687 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479691 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479694 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479697 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479700 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 05:53:53.484279 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479703 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479705 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479708 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479711 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479715 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479717 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479720 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479722 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479725 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479727 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479729 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479732 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479734 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479737 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479739 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479742 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479744 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479747 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479749 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479752 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 05:53:53.484738 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479756 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.479758 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479831 2580 flags.go:64] FLAG: --address="0.0.0.0" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479863 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479869 2580 flags.go:64] FLAG: --anonymous-auth="true" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479873 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479877 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479881 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479885 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479889 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479892 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479895 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479899 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479902 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479905 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479909 2580 flags.go:64] FLAG: --cgroup-root="" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479913 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479916 2580 flags.go:64] FLAG: --client-ca-file="" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479918 2580 flags.go:64] FLAG: --cloud-provider="external" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479921 2580 flags.go:64] FLAG: --cluster-dns="[]" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479926 2580 flags.go:64] FLAG: --cluster-domain="" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479928 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479931 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479934 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 06 05:53:53.485241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479937 2580 flags.go:64] FLAG: --container-log-max-files="5" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479941 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479944 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479947 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479950 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479952 2580 flags.go:64] FLAG: --contention-profiling="false" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479955 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479958 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479962 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479965 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479969 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479972 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479975 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479977 2580 flags.go:64] FLAG: --enable-load-reader="false" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479980 2580 flags.go:64] FLAG: --enable-server="true" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479983 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479987 2580 flags.go:64] FLAG: --event-burst="100" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479990 2580 flags.go:64] FLAG: --event-qps="50" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479993 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479996 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.479998 2580 flags.go:64] FLAG: --eviction-hard="" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480002 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480005 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480008 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480012 2580 flags.go:64] FLAG: --eviction-soft="" Sep 06 05:53:53.485819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480014 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480017 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480020 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480023 2580 flags.go:64] FLAG: --experimental-mounter-path="" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480026 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480029 2580 flags.go:64] FLAG: --fail-swap-on="true" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480031 2580 flags.go:64] FLAG: --feature-gates="" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480035 2580 flags.go:64] FLAG: --file-check-frequency="20s" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480039 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480056 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480062 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480066 2580 flags.go:64] FLAG: --healthz-port="10248" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480071 2580 flags.go:64] FLAG: --help="false" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480075 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480078 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480082 2580 flags.go:64] FLAG: --http-check-frequency="20s" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480085 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480088 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480092 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480095 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480097 2580 flags.go:64] FLAG: --image-service-endpoint="" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480100 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480103 2580 flags.go:64] FLAG: --kube-api-burst="100" Sep 06 05:53:53.486430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480106 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480109 2580 flags.go:64] FLAG: --kube-api-qps="50" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480111 2580 flags.go:64] FLAG: --kube-reserved="" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480114 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480117 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480120 2580 flags.go:64] FLAG: --kubelet-cgroups="" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480123 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480126 2580 flags.go:64] FLAG: --lock-file="" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480129 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480133 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480136 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480142 2580 flags.go:64] FLAG: --log-json-split-stream="false" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480144 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480147 2580 flags.go:64] FLAG: --log-text-split-stream="false" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480150 2580 flags.go:64] FLAG: --logging-format="text" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480153 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480156 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480159 2580 flags.go:64] FLAG: --manifest-url="" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480162 2580 flags.go:64] FLAG: --manifest-url-header="" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480166 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480169 2580 flags.go:64] FLAG: --max-open-files="1000000" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480173 2580 flags.go:64] FLAG: --max-pods="110" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480176 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480179 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480182 2580 flags.go:64] FLAG: --memory-manager-policy="None" Sep 06 05:53:53.487084 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480185 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480188 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480191 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480194 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480201 2580 flags.go:64] FLAG: --node-status-max-images="50" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480204 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480207 2580 flags.go:64] FLAG: --oom-score-adj="-999" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480210 2580 flags.go:64] FLAG: --pod-cidr="" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480213 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480218 2580 flags.go:64] FLAG: --pod-manifest-path="" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480221 2580 flags.go:64] FLAG: --pod-max-pids="-1" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480224 2580 flags.go:64] FLAG: --pods-per-core="0" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480227 2580 flags.go:64] FLAG: --port="10250" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480230 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480233 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f06ea9a67a9e577b" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480236 2580 flags.go:64] FLAG: --qos-reserved="" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480239 2580 flags.go:64] FLAG: --read-only-port="10255" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480242 2580 flags.go:64] FLAG: --register-node="true" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480245 2580 flags.go:64] FLAG: --register-with-taints="" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480249 2580 flags.go:64] FLAG: --registry-burst="10" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480251 2580 flags.go:64] FLAG: --registry-qps="5" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480254 2580 flags.go:64] FLAG: --reserved-cpus="" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480258 2580 flags.go:64] FLAG: --reserved-memory="" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480261 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480264 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 06 05:53:53.487703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480267 2580 flags.go:64] FLAG: --rotate-certificates="false" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480270 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480272 2580 flags.go:64] FLAG: --runonce="false" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480276 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480279 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480281 2580 flags.go:64] FLAG: --seccomp-default="false" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480284 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480287 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480290 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480293 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480296 2580 flags.go:64] FLAG: --storage-driver-password="root" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480299 2580 flags.go:64] FLAG: --storage-driver-secure="false" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480302 2580 flags.go:64] FLAG: --storage-driver-table="stats" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480305 2580 flags.go:64] FLAG: --storage-driver-user="root" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480308 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480311 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480313 2580 flags.go:64] FLAG: --system-cgroups="" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480316 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480322 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480324 2580 flags.go:64] FLAG: --tls-cert-file="" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480327 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480333 2580 flags.go:64] FLAG: --tls-min-version="" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480336 2580 flags.go:64] FLAG: --tls-private-key-file="" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480339 2580 flags.go:64] FLAG: --topology-manager-policy="none" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480342 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 06 05:53:53.488329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480345 2580 flags.go:64] FLAG: --topology-manager-scope="container" Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480348 2580 flags.go:64] FLAG: --v="2" Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480352 2580 flags.go:64] FLAG: --version="false" Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480361 2580 flags.go:64] FLAG: --vmodule="" Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480366 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480369 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480479 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480483 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480486 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480489 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480492 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480495 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480498 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480501 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480504 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480507 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480510 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480513 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480529 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480533 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480536 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 05:53:53.488934 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480539 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480542 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480545 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480547 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480550 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480553 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480555 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480560 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480562 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480565 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480567 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480570 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480574 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480577 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480579 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480583 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480586 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480589 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480591 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480594 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 05:53:53.489538 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480596 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480599 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480601 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480604 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480606 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480609 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480612 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480614 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480617 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480620 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480624 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480627 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480631 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480633 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480636 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480638 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480641 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480643 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480646 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 05:53:53.490058 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480651 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480653 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480656 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480659 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480662 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480664 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480667 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480670 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480674 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480677 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480680 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480682 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480684 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480687 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480689 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480692 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480696 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480699 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480702 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480705 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 05:53:53.490541 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480707 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480710 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480712 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480715 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480717 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480720 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480722 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480725 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480727 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480730 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480732 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480735 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480740 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480743 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480746 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480749 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480752 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480754 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480756 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 05:53:53.491153 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480762 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480766 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480769 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480772 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480774 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.480777 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.480781 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.488251 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.488266 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.489155 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.489167 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489221 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489227 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489230 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489234 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 05:53:53.491647 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489236 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489239 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489242 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489244 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489247 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489251 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489269 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489273 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489276 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489279 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489283 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489286 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489288 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489291 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489293 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489296 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489299 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489301 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489304 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 05:53:53.492024 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489306 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489309 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489312 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489314 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489317 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489320 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489324 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489327 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489329 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489332 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489334 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489337 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489340 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489342 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489345 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489347 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489350 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489353 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489355 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489358 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 05:53:53.492496 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489360 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489363 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489365 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489369 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489372 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489375 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489378 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489380 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489383 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489386 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489388 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489391 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489393 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489395 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489398 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489400 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489403 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489406 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489409 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489413 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 05:53:53.492982 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489415 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489418 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489421 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489423 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489426 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489428 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489431 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489433 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489436 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489439 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489442 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489446 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489449 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489451 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489454 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489457 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489460 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489463 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489466 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 05:53:53.493469 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489469 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489472 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489474 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489477 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489480 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489483 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489485 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489488 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489490 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489493 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489495 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489498 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489501 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489504 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489507 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489509 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 05:53:53.493937 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489512 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.489517 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489623 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489628 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489631 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489634 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489636 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489639 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489642 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489645 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489648 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489650 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489653 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489657 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489659 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 05:53:53.494371 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489662 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489665 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489668 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489670 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489673 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489675 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489678 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489681 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489683 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489685 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489688 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489691 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489694 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489697 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489700 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489703 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489706 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489708 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489712 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 05:53:53.494749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489716 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489720 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489723 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489725 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489728 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489731 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489734 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489737 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489739 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489742 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489745 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489747 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489750 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489753 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489755 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489757 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489760 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489762 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489765 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489767 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 05:53:53.495226 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489770 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489772 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489775 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489777 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489780 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489783 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489785 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489788 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489790 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489793 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489796 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489798 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489801 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489803 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489806 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489808 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489811 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489813 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489816 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489818 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 05:53:53.495749 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489821 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489823 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489826 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489828 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489831 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489833 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489836 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489838 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489840 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489843 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489845 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489848 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489850 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489853 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489855 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489857 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489860 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489863 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489865 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489868 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 05:53:53.496246 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489870 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 05:53:53.496751 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489873 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 05:53:53.496751 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489876 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 05:53:53.496751 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489878 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 05:53:53.496751 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489881 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 05:53:53.496751 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489884 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 05:53:53.496751 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:53.489887 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 05:53:53.496751 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.489892 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 05:53:53.496751 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.489990 2580 server.go:962] "Client rotation is on, will bootstrap in background" Sep 06 05:53:53.496751 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.492225 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 06 05:53:53.496751 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.493221 2580 server.go:1019] "Starting client certificate rotation" Sep 06 05:53:53.496751 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.493323 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 06 05:53:53.496751 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.493352 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 06 05:53:53.497037 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.496755 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 06 05:53:53.497989 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.497967 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 06 05:53:53.509319 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.509254 2580 log.go:25] "Validated CRI v1 runtime API" Sep 06 05:53:53.513378 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.513361 2580 log.go:25] "Validated CRI v1 image API" Sep 06 05:53:53.514161 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.514147 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 06 05:53:53.517460 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.517442 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cf9b4614-6caf-4370-866d-23f9e6a69def:/dev/nvme0n1p4 fdd02c1b-5840-4d51-a0df-f6d4ae86d4a0:/dev/nvme0n1p3] Sep 06 05:53:53.517515 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.517458 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 06 05:53:53.520104 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.520087 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 06 05:53:53.523900 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.523787 2580 manager.go:216] Machine: {Timestamp:2026-09-06 05:53:53.522388145 +0000 UTC m=+0.370791286 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3129961 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29b2717e9ede51f29aca519a1f8309 SystemUUID:ec29b271-7e9e-de51-f29a-ca519a1f8309 BootID:8e0b3a6b-a166-4a62-8d08-e62d128dff85 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:12:e4:33:6b:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:12:e4:33:6b:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:31:ce:85:5a:1d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 06 05:53:53.523900 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.523895 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 06 05:53:53.524019 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.523970 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 06 05:53:53.527548 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.527526 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 06 05:53:53.527696 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.527552 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-34.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 06 05:53:53.529090 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.529081 2580 topology_manager.go:138] "Creating topology manager with none policy" Sep 06 05:53:53.529123 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.529093 2580 container_manager_linux.go:309] "Creating device plugin manager" Sep 06 05:53:53.529123 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.529102 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 06 05:53:53.529123 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.529116 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 06 05:53:53.531012 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.531001 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 06 05:53:53.535285 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.535274 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 06 05:53:53.535400 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.535391 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 06 05:53:53.535433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.535426 2580 kubelet.go:486] "Attempting to sync node with API server" Sep 06 05:53:53.535463 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.535435 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 06 05:53:53.535463 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.535447 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 06 05:53:53.535463 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.535457 2580 kubelet.go:398] "Adding apiserver pod source" Sep 06 05:53:53.535555 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.535465 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 06 05:53:53.535839 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.535827 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 06 05:53:53.535895 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.535887 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 06 05:53:53.538778 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.538755 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 06 05:53:53.541436 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.541421 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 06 05:53:53.541500 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.541451 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 06 05:53:53.541619 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.541610 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 06 05:53:53.541653 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.541625 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 06 05:53:53.541653 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.541631 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 06 05:53:53.541653 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.541637 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 06 05:53:53.541653 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.541647 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 06 05:53:53.541785 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.541654 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 06 05:53:53.542173 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.542164 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 06 05:53:53.542173 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.542173 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 06 05:53:53.542248 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.542179 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 06 05:53:53.542248 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.542184 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 06 05:53:53.542248 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.542192 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 06 05:53:53.542248 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.542200 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 06 05:53:53.543078 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.543066 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 06 05:53:53.543078 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.543076 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 06 05:53:53.545459 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.545440 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-34.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 06 05:53:53.546244 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.546219 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 06 05:53:53.546304 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.546218 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 06 05:53:53.547053 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.547028 2580 server.go:1268] "Started kubelet" Sep 06 05:53:53.547147 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.547108 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 06 05:53:53.547899 ip-10-0-142-34 systemd[1]: Started Kubernetes Kubelet. Sep 06 05:53:53.548111 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.548066 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 06 05:53:53.548166 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.548141 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 06 05:53:53.551636 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.551607 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 06 05:53:53.552706 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.552688 2580 server.go:310] "Adding debug handlers to kubelet server" Sep 06 05:53:53.554716 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.554688 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 06 05:53:53.554934 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.553869 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-34.ec2.internal.18d2a58e8488b306 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-34.ec2.internal,UID:ip-10-0-142-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-34.ec2.internal,},FirstTimestamp:2026-09-06 05:53:53.546998534 +0000 UTC m=+0.395401673,LastTimestamp:2026-09-06 05:53:53.546998534 +0000 UTC m=+0.395401673,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-34.ec2.internal,}" Sep 06 05:53:53.557089 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.557070 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 06 05:53:53.557781 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.557767 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 06 05:53:53.557931 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.557912 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:53.558519 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.558503 2580 factory.go:55] Registering systemd factory Sep 06 05:53:53.558519 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.558519 2580 factory.go:223] Registration of the systemd container factory successfully Sep 06 05:53:53.558639 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.558599 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 06 05:53:53.558639 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.558600 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 06 05:53:53.558639 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.558622 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 06 05:53:53.558738 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.558725 2580 reconstruct.go:97] "Volume reconstruction finished" Sep 06 05:53:53.558738 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.558735 2580 reconciler.go:29] "Reconciler: start to sync state" Sep 06 05:53:53.558795 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.558742 2580 factory.go:153] Registering CRI-O factory Sep 06 05:53:53.558795 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.558758 2580 factory.go:223] Registration of the crio container factory successfully Sep 06 05:53:53.558861 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.558819 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 06 05:53:53.558861 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.558835 2580 factory.go:103] Registering Raw factory Sep 06 05:53:53.558861 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.558846 2580 manager.go:1196] Started watching for new ooms in manager Sep 06 05:53:53.559241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.559227 2580 manager.go:349] Starting recovery of all containers Sep 06 05:53:53.559951 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.559806 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 06 05:53:53.560109 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.560092 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 06 05:53:53.566402 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.566346 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 06 05:53:53.566525 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.566503 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7g8ft" Sep 06 05:53:53.569353 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.569334 2580 manager.go:354] Recovery completed Sep 06 05:53:53.572363 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.572327 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7g8ft" Sep 06 05:53:53.574882 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.574867 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.576626 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.576611 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.576696 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.576636 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.576696 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.576646 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.577228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.577204 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 06 05:53:53.577228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.577218 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 06 05:53:53.577344 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.577232 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.577344 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.577235 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 06 05:53:53.577911 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.577888 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.577981 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.577916 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.577981 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.577932 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.579172 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.579155 2580 policy_none.go:49] "None policy: Start" Sep 06 05:53:53.579237 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.579176 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 06 05:53:53.579237 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.579197 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 06 05:53:53.581100 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.581086 2580 policy_none.go:47] "Start" Sep 06 05:53:53.618660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.618641 2580 manager.go:337] "Starting Device Plugin manager" Sep 06 05:53:53.642990 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.618684 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 06 05:53:53.642990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.618698 2580 server.go:87] "Starting device plugin registration server" Sep 06 05:53:53.642990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.619081 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 06 05:53:53.642990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.619094 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 06 05:53:53.642990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.619190 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 06 05:53:53.642990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.619262 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 06 05:53:53.642990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.619270 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 06 05:53:53.642990 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.620602 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 06 05:53:53.642990 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.620644 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:53.668499 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.668474 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 06 05:53:53.668584 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.668532 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 06 05:53:53.668584 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.668560 2580 kubelet.go:2448] "Starting kubelet main sync loop" Sep 06 05:53:53.668660 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.668605 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 06 05:53:53.670598 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.670580 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 05:53:53.719397 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.719362 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.720372 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.720355 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.720458 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.720385 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.720458 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.720395 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.720458 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.720419 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.729203 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.729138 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.729203 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.729165 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-34.ec2.internal\": node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:53.753608 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.753578 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:53.768900 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.768872 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-34.ec2.internal"] Sep 06 05:53:53.768991 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.768943 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.769827 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.769810 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.769913 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.769839 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.769913 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.769852 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.771975 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.771964 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.772119 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.772099 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.772160 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.772131 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.772748 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.772726 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.772831 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.772759 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.772831 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.772771 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.772831 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.772729 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.772831 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.772828 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.772953 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.772841 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.774966 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.774950 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.775076 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.774974 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.775686 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.775670 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.775780 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.775693 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.775780 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.775702 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.803922 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.803893 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-34.ec2.internal\" not found" node="ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.808344 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.808326 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-34.ec2.internal\" not found" node="ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.854020 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.853998 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:53.859805 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.859786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e13a78121acfe93071c1c9eb406a02ec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal\" (UID: \"e13a78121acfe93071c1c9eb406a02ec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.859853 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.859817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e13a78121acfe93071c1c9eb406a02ec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal\" (UID: \"e13a78121acfe93071c1c9eb406a02ec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.859853 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.859836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b5baafa68872d0e1a89c3b7fe397e3a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-34.ec2.internal\" (UID: \"6b5baafa68872d0e1a89c3b7fe397e3a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.954354 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:53.954329 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:53.960670 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.960656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e13a78121acfe93071c1c9eb406a02ec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal\" (UID: \"e13a78121acfe93071c1c9eb406a02ec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.960732 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.960718 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e13a78121acfe93071c1c9eb406a02ec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal\" (UID: \"e13a78121acfe93071c1c9eb406a02ec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.960772 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.960739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e13a78121acfe93071c1c9eb406a02ec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal\" (UID: \"e13a78121acfe93071c1c9eb406a02ec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.960772 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.960766 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b5baafa68872d0e1a89c3b7fe397e3a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-34.ec2.internal\" (UID: \"6b5baafa68872d0e1a89c3b7fe397e3a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.960868 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.960807 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b5baafa68872d0e1a89c3b7fe397e3a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-34.ec2.internal\" (UID: \"6b5baafa68872d0e1a89c3b7fe397e3a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-34.ec2.internal" Sep 06 05:53:53.960868 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:53.960850 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e13a78121acfe93071c1c9eb406a02ec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal\" (UID: \"e13a78121acfe93071c1c9eb406a02ec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" Sep 06 05:53:54.055020 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:54.054964 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:54.106473 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:54.106446 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" Sep 06 05:53:54.111665 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:54.111016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-34.ec2.internal" Sep 06 05:53:54.155509 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:54.155480 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:54.256023 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:54.255994 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:54.356591 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:54.356526 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:55.656821 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:55.656795 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:55.693260 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:55.693223 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 06 05:53:55.693620 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:55.693391 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 06 05:53:55.757181 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:55.757156 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:55.775021 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:55.774987 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-05 05:48:54 +0000 UTC" deadline="2028-06-14 15:00:56.957748888 +0000 UTC" Sep 06 05:53:55.775021 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:55.775016 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15537h7m1.182737043s" Sep 06 05:53:55.792062 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:55.792042 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 05:53:55.857346 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:55.857271 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-34.ec2.internal\" not found" Sep 06 05:53:55.864213 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:55.864195 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 05:53:55.912189 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:55.912167 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 05:53:55.958429 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:55.958400 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" Sep 06 05:53:55.970875 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:55.970854 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 06 05:53:55.971940 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:55.971927 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-34.ec2.internal" Sep 06 05:53:55.981801 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:55.981781 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 06 05:53:56.185025 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:56.184989 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5baafa68872d0e1a89c3b7fe397e3a.slice/crio-7d5ed8e72d56e1e655effd1eeac91ddd75df7f6b17e0591f285b5be857923882 WatchSource:0}: Error finding container 7d5ed8e72d56e1e655effd1eeac91ddd75df7f6b17e0591f285b5be857923882: Status 404 returned error can't find the container with id 7d5ed8e72d56e1e655effd1eeac91ddd75df7f6b17e0591f285b5be857923882 Sep 06 05:53:56.185479 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:56.185458 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13a78121acfe93071c1c9eb406a02ec.slice/crio-f814a12c64060f7c1817d6b21242039e372ec5c1ac8f7951403f551f962339ab WatchSource:0}: Error finding container f814a12c64060f7c1817d6b21242039e372ec5c1ac8f7951403f551f962339ab: Status 404 returned error can't find the container with id f814a12c64060f7c1817d6b21242039e372ec5c1ac8f7951403f551f962339ab Sep 06 05:53:56.190163 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.190147 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 05:53:56.594589 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.594512 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 05:53:56.735754 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.735714 2580 apiserver.go:52] "Watching apiserver" Sep 06 05:53:56.744125 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.744094 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 06 05:53:56.747070 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.747034 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-cb5np","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal","openshift-multus/network-metrics-daemon-2z6qh","openshift-network-diagnostics/network-check-target-jlt55","openshift-ovn-kubernetes/ovnkube-node-84rs9","kube-system/global-pull-secret-syncer-nksgl","kube-system/konnectivity-agent-dmd4j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2","openshift-dns/node-resolver-xxk4z","openshift-multus/multus-additional-cni-plugins-cqfs6","openshift-multus/multus-gjhdl","openshift-network-operator/iptables-alerter-8bqn2","kube-system/kube-apiserver-proxy-ip-10-0-142-34.ec2.internal","openshift-cluster-node-tuning-operator/tuned-vrsf2"] Sep 06 05:53:56.750897 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.750880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.752874 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.752854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.753350 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.753328 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 06 05:53:56.753447 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.753328 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 06 05:53:56.753590 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.753571 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 06 05:53:56.753661 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.753632 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sqcw2\"" Sep 06 05:53:56.753713 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.753632 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 06 05:53:56.753761 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.753714 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 06 05:53:56.755004 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.754985 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 06 05:53:56.755099 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.755022 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bjcz8\"" Sep 06 05:53:56.755099 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.755069 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.757331 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.757125 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 06 05:53:56.757331 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.757246 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9qfq8\"" Sep 06 05:53:56.757331 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.757270 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 06 05:53:56.757545 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.757528 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 06 05:53:56.757651 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.757634 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 06 05:53:56.757795 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.757768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:53:56.757955 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:56.757928 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:53:56.760059 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.760040 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.762487 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.762471 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:53:56.762560 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:56.762538 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:53:56.764624 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.764601 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 06 05:53:56.764706 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.764641 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 06 05:53:56.764706 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.764650 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r2fj7\"" Sep 06 05:53:56.764706 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.764655 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 06 05:53:56.764706 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.764691 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 06 05:53:56.764943 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.764903 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xxk4z" Sep 06 05:53:56.764998 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.764962 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 06 05:53:56.765050 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.765006 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 06 05:53:56.767420 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.767402 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z45b4\"" Sep 06 05:53:56.767550 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.767534 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 06 05:53:56.767707 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.767657 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cb5np" Sep 06 05:53:56.767707 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.767700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-slash\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.767819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.767752 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-run-netns\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.767819 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.767782 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 06 05:53:56.767940 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.767785 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-node-log\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.767940 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.767874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-cnibin\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.767940 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.767899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-var-lib-kubelet\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.767940 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.767921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-run-multus-certs\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.768133 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.767957 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gns46\" (UniqueName: \"kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46\") pod \"network-check-target-jlt55\" (UID: \"32d81600-7fda-434c-8df0-e43a6c3bd84c\") " pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:53:56.768133 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.767997 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-run-systemd\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768133 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-etc-openvswitch\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768133 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-cni-netd\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768133 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768073 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-env-overrides\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768133 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768099 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkjbr\" (UniqueName: \"kubernetes.io/projected/08082397-d99f-4ed1-af18-c48acdeb20e0-kube-api-access-nkjbr\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-hostroot\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768204 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-system-cni-dir\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-multus-socket-dir-parent\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-run-netns\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-etc-kubernetes\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768285 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-var-lib-openvswitch\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-run-ovn\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-sys-fs\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-systemd-units\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-ovnkube-script-lib\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768390 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/678e8aa3-c088-4f39-bf3d-f158f45c9aef-cnibin\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/678e8aa3-c088-4f39-bf3d-f158f45c9aef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.768433 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/678e8aa3-c088-4f39-bf3d-f158f45c9aef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl2zm\" (UniqueName: \"kubernetes.io/projected/678e8aa3-c088-4f39-bf3d-f158f45c9aef-kube-api-access-sl2zm\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95h28\" (UniqueName: \"kubernetes.io/projected/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-kube-api-access-95h28\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-run-openvswitch\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-log-socket\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/678e8aa3-c088-4f39-bf3d-f158f45c9aef-system-cni-dir\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/678e8aa3-c088-4f39-bf3d-f158f45c9aef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-multus-daemon-config\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/678e8aa3-c088-4f39-bf3d-f158f45c9aef-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-registration-dir\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-device-dir\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th6cm\" (UniqueName: \"kubernetes.io/projected/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-kube-api-access-th6cm\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.768990 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-run-k8s-cni-cncf-io\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.769649 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.768990 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-var-lib-cni-bin\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.769649 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.769380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-var-lib-cni-multus\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.769649 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.769394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-multus-conf-dir\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.769649 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.769407 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-socket-dir\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.769649 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.769420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-cni-bin\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.769649 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.769435 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-ovnkube-config\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.769649 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.769450 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-ovn-node-metrics-cert\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.769649 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.769472 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-multus-cni-dir\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.769649 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.769493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-cni-binary-copy\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.769649 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.769510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-kubelet\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.770227 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.770060 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k5xwn\"" Sep 06 05:53:56.770227 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.770089 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 06 05:53:56.770338 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.770258 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dmd4j" Sep 06 05:53:56.770338 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.770298 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 06 05:53:56.770426 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.770358 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 06 05:53:56.770554 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.770527 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 06 05:53:56.772638 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.772619 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2z66f\"" Sep 06 05:53:56.772715 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.772648 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8bqn2" Sep 06 05:53:56.772778 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.772726 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 06 05:53:56.773234 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.773219 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 06 05:53:56.775971 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.775297 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:53:56.775971 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:56.775403 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:53:56.775971 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.775466 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 06 05:53:56.775971 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.775496 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 06 05:53:56.775971 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.775938 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 06 05:53:56.776317 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.776223 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v9d7h\"" Sep 06 05:53:56.779860 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.779817 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.782372 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.782352 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 06 05:53:56.782521 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.782507 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wtxjc\"" Sep 06 05:53:56.783375 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.783358 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 06 05:53:56.785715 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.785701 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5cmgn" Sep 06 05:53:56.808452 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.808437 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5cmgn" Sep 06 05:53:56.859175 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.859110 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 06 05:53:56.869735 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.869704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2zm\" (UniqueName: \"kubernetes.io/projected/678e8aa3-c088-4f39-bf3d-f158f45c9aef-kube-api-access-sl2zm\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.869862 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.869763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:53:56.869933 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.869857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcszm\" (UniqueName: \"kubernetes.io/projected/e5ada9e9-c4df-486d-9325-b5dcecdd451b-kube-api-access-rcszm\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:53:56.869933 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.869888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-cnibin\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.870042 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.869945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-run-systemd\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.870042 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.869977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/678e8aa3-c088-4f39-bf3d-f158f45c9aef-system-cni-dir\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.870042 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkjbr\" (UniqueName: \"kubernetes.io/projected/08082397-d99f-4ed1-af18-c48acdeb20e0-kube-api-access-nkjbr\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.870042 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-hostroot\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.870228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870058 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-cnibin\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.870228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-system-cni-dir\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.870228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870074 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/678e8aa3-c088-4f39-bf3d-f158f45c9aef-system-cni-dir\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.870228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-etc-kubernetes\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.870228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-run-ovn\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.870228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27b14080-015b-496f-b375-cc4cbdbf1892-hosts-file\") pod \"node-resolver-xxk4z\" (UID: \"27b14080-015b-496f-b375-cc4cbdbf1892\") " pod="openshift-dns/node-resolver-xxk4z" Sep 06 05:53:56.870228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.870228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870164 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-hostroot\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.870228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/678e8aa3-c088-4f39-bf3d-f158f45c9aef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.870228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef0b0f61-4619-4c40-afd3-0a83e4e2808a-host-slash\") pod \"iptables-alerter-8bqn2\" (UID: \"ef0b0f61-4619-4c40-afd3-0a83e4e2808a\") " pod="openshift-network-operator/iptables-alerter-8bqn2" Sep 06 05:53:56.870228 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870238 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-kubernetes\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-sysctl-d\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870287 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-sysctl-conf\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sgj8\" (UniqueName: \"kubernetes.io/projected/c80ff024-ecdd-468b-a518-167183800b69-kube-api-access-2sgj8\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870130 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-run-systemd\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870374 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-etc-kubernetes\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-run-openvswitch\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-system-cni-dir\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-log-socket\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870526 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-run-ovn\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-run-openvswitch\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/678e8aa3-c088-4f39-bf3d-f158f45c9aef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870625 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-log-socket\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870627 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbqc4\" (UniqueName: \"kubernetes.io/projected/ef0b0f61-4619-4c40-afd3-0a83e4e2808a-kube-api-access-tbqc4\") pod \"iptables-alerter-8bqn2\" (UID: \"ef0b0f61-4619-4c40-afd3-0a83e4e2808a\") " pod="openshift-network-operator/iptables-alerter-8bqn2" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-multus-daemon-config\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.870784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870748 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce82693f-a593-41cf-83c0-d09f07d50168-serviceca\") pod \"node-ca-cb5np\" (UID: \"ce82693f-a593-41cf-83c0-d09f07d50168\") " pod="openshift-image-registry/node-ca-cb5np" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870772 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/678e8aa3-c088-4f39-bf3d-f158f45c9aef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/81e0c9ba-7535-45d5-933f-4aa8c33194c0-konnectivity-ca\") pod \"konnectivity-agent-dmd4j\" (UID: \"81e0c9ba-7535-45d5-933f-4aa8c33194c0\") " pod="kube-system/konnectivity-agent-dmd4j" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-var-lib-kubelet\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-device-dir\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th6cm\" (UniqueName: \"kubernetes.io/projected/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-kube-api-access-th6cm\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b14080-015b-496f-b375-cc4cbdbf1892-tmp-dir\") pod \"node-resolver-xxk4z\" (UID: \"27b14080-015b-496f-b375-cc4cbdbf1892\") " pod="openshift-dns/node-resolver-xxk4z" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-modprobe-d\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871022 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-device-dir\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.870994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c80ff024-ecdd-468b-a518-167183800b69-etc-tuned\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-run-k8s-cni-cncf-io\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-var-lib-cni-bin\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-var-lib-cni-multus\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-multus-conf-dir\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-ovn-node-metrics-cert\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871239 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-var-lib-cni-bin\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.871537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-var-lib-cni-multus\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-run-k8s-cni-cncf-io\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-multus-cni-dir\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871341 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-multus-conf-dir\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-kubelet\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-slash\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-node-log\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-multus-cni-dir\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-env-overrides\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e95c4ec-ee79-4651-9f84-6f1bd6660624-dbus\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871547 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjjv\" (UniqueName: \"kubernetes.io/projected/27b14080-015b-496f-b375-cc4cbdbf1892-kube-api-access-7kjjv\") pod \"node-resolver-xxk4z\" (UID: \"27b14080-015b-496f-b375-cc4cbdbf1892\") " pod="openshift-dns/node-resolver-xxk4z" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-kubelet\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-var-lib-kubelet\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-node-log\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-run-multus-certs\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871620 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/678e8aa3-c088-4f39-bf3d-f158f45c9aef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871685 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-slash\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871650 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-var-lib-kubelet\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.872293 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871728 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-run-multus-certs\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns46\" (UniqueName: \"kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46\") pod \"network-check-target-jlt55\" (UID: \"32d81600-7fda-434c-8df0-e43a6c3bd84c\") " pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-etc-openvswitch\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-cni-netd\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-etc-openvswitch\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871946 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjgtc\" (UniqueName: \"kubernetes.io/projected/ce82693f-a593-41cf-83c0-d09f07d50168-kube-api-access-tjgtc\") pod \"node-ca-cb5np\" (UID: \"ce82693f-a593-41cf-83c0-d09f07d50168\") " pod="openshift-image-registry/node-ca-cb5np" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871965 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-multus-daemon-config\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.871978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c80ff024-ecdd-468b-a518-167183800b69-tmp\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-multus-socket-dir-parent\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872030 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-cni-netd\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-run-netns\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-var-lib-openvswitch\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-multus-socket-dir-parent\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872178 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-host-run-netns\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/678e8aa3-c088-4f39-bf3d-f158f45c9aef-cnibin\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872217 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-env-overrides\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872223 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-var-lib-openvswitch\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.873188 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/678e8aa3-c088-4f39-bf3d-f158f45c9aef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e95c4ec-ee79-4651-9f84-6f1bd6660624-kubelet-config\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872386 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/678e8aa3-c088-4f39-bf3d-f158f45c9aef-cnibin\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-sys-fs\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-systemd-units\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-ovnkube-script-lib\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872565 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-systemd\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-sys\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-lib-modules\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95h28\" (UniqueName: \"kubernetes.io/projected/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-kube-api-access-95h28\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef0b0f61-4619-4c40-afd3-0a83e4e2808a-iptables-alerter-script\") pod \"iptables-alerter-8bqn2\" (UID: \"ef0b0f61-4619-4c40-afd3-0a83e4e2808a\") " pod="openshift-network-operator/iptables-alerter-8bqn2" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872720 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-sysconfig\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872713 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/678e8aa3-c088-4f39-bf3d-f158f45c9aef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872802 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/678e8aa3-c088-4f39-bf3d-f158f45c9aef-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.872843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-run\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.874033 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873120 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873201 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-ovnkube-script-lib\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873238 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-sys-fs\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/678e8aa3-c088-4f39-bf3d-f158f45c9aef-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-registration-dir\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/81e0c9ba-7535-45d5-933f-4aa8c33194c0-agent-certs\") pod \"konnectivity-agent-dmd4j\" (UID: \"81e0c9ba-7535-45d5-933f-4aa8c33194c0\") " pod="kube-system/konnectivity-agent-dmd4j" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-systemd-units\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-registration-dir\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-host\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-socket-dir\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873560 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-cni-bin\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-ovnkube-config\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce82693f-a593-41cf-83c0-d09f07d50168-host\") pod \"node-ca-cb5np\" (UID: \"ce82693f-a593-41cf-83c0-d09f07d50168\") " pod="openshift-image-registry/node-ca-cb5np" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-cni-binary-copy\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873647 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08082397-d99f-4ed1-af18-c48acdeb20e0-socket-dir\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.874808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-cni-bin\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.875571 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-run-netns\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.875571 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.873740 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-host-run-netns\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.875571 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.874173 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-ovnkube-config\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.875571 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.874238 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-cni-binary-copy\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.875571 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.874461 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-34.ec2.internal" event={"ID":"6b5baafa68872d0e1a89c3b7fe397e3a","Type":"ContainerStarted","Data":"7d5ed8e72d56e1e655effd1eeac91ddd75df7f6b17e0591f285b5be857923882"} Sep 06 05:53:56.876484 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.876462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-ovn-node-metrics-cert\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.876654 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.876629 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" event={"ID":"e13a78121acfe93071c1c9eb406a02ec","Type":"ContainerStarted","Data":"f814a12c64060f7c1817d6b21242039e372ec5c1ac8f7951403f551f962339ab"} Sep 06 05:53:56.877927 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:56.877875 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:53:56.877927 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:56.877902 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:53:56.877927 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:56.877915 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gns46 for pod openshift-network-diagnostics/network-check-target-jlt55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:53:56.878111 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:56.878007 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46 podName:32d81600-7fda-434c-8df0-e43a6c3bd84c nodeName:}" failed. No retries permitted until 2026-09-06 05:53:57.377982236 +0000 UTC m=+3.026601393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gns46" (UniqueName: "kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46") pod "network-check-target-jlt55" (UID: "32d81600-7fda-434c-8df0-e43a6c3bd84c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:53:56.880114 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.880091 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2zm\" (UniqueName: \"kubernetes.io/projected/678e8aa3-c088-4f39-bf3d-f158f45c9aef-kube-api-access-sl2zm\") pod \"multus-additional-cni-plugins-cqfs6\" (UID: \"678e8aa3-c088-4f39-bf3d-f158f45c9aef\") " pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:56.880567 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.880546 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkjbr\" (UniqueName: \"kubernetes.io/projected/08082397-d99f-4ed1-af18-c48acdeb20e0-kube-api-access-nkjbr\") pod \"aws-ebs-csi-driver-node-nvhv2\" (UID: \"08082397-d99f-4ed1-af18-c48acdeb20e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:56.881657 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.881375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th6cm\" (UniqueName: \"kubernetes.io/projected/dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9-kube-api-access-th6cm\") pod \"ovnkube-node-84rs9\" (UID: \"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:56.881657 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.881495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95h28\" (UniqueName: \"kubernetes.io/projected/513bdbba-c5b6-4ea9-80ea-f909cbba8abb-kube-api-access-95h28\") pod \"multus-gjhdl\" (UID: \"513bdbba-c5b6-4ea9-80ea-f909cbba8abb\") " pod="openshift-multus/multus-gjhdl" Sep 06 05:53:56.974081 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef0b0f61-4619-4c40-afd3-0a83e4e2808a-host-slash\") pod \"iptables-alerter-8bqn2\" (UID: \"ef0b0f61-4619-4c40-afd3-0a83e4e2808a\") " pod="openshift-network-operator/iptables-alerter-8bqn2" Sep 06 05:53:56.974259 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-kubernetes\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974259 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-sysctl-d\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974259 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-sysctl-conf\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974259 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sgj8\" (UniqueName: \"kubernetes.io/projected/c80ff024-ecdd-468b-a518-167183800b69-kube-api-access-2sgj8\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974259 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbqc4\" (UniqueName: \"kubernetes.io/projected/ef0b0f61-4619-4c40-afd3-0a83e4e2808a-kube-api-access-tbqc4\") pod \"iptables-alerter-8bqn2\" (UID: \"ef0b0f61-4619-4c40-afd3-0a83e4e2808a\") " pod="openshift-network-operator/iptables-alerter-8bqn2" Sep 06 05:53:56.974259 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce82693f-a593-41cf-83c0-d09f07d50168-serviceca\") pod \"node-ca-cb5np\" (UID: \"ce82693f-a593-41cf-83c0-d09f07d50168\") " pod="openshift-image-registry/node-ca-cb5np" Sep 06 05:53:56.974259 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/81e0c9ba-7535-45d5-933f-4aa8c33194c0-konnectivity-ca\") pod \"konnectivity-agent-dmd4j\" (UID: \"81e0c9ba-7535-45d5-933f-4aa8c33194c0\") " pod="kube-system/konnectivity-agent-dmd4j" Sep 06 05:53:56.974259 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974245 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-var-lib-kubelet\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b14080-015b-496f-b375-cc4cbdbf1892-tmp-dir\") pod \"node-resolver-xxk4z\" (UID: \"27b14080-015b-496f-b375-cc4cbdbf1892\") " pod="openshift-dns/node-resolver-xxk4z" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-modprobe-d\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c80ff024-ecdd-468b-a518-167183800b69-etc-tuned\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e95c4ec-ee79-4651-9f84-6f1bd6660624-dbus\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjjv\" (UniqueName: \"kubernetes.io/projected/27b14080-015b-496f-b375-cc4cbdbf1892-kube-api-access-7kjjv\") pod \"node-resolver-xxk4z\" (UID: \"27b14080-015b-496f-b375-cc4cbdbf1892\") " pod="openshift-dns/node-resolver-xxk4z" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjgtc\" (UniqueName: \"kubernetes.io/projected/ce82693f-a593-41cf-83c0-d09f07d50168-kube-api-access-tjgtc\") pod \"node-ca-cb5np\" (UID: \"ce82693f-a593-41cf-83c0-d09f07d50168\") " pod="openshift-image-registry/node-ca-cb5np" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c80ff024-ecdd-468b-a518-167183800b69-tmp\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e95c4ec-ee79-4651-9f84-6f1bd6660624-kubelet-config\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-systemd\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-sys\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-lib-modules\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef0b0f61-4619-4c40-afd3-0a83e4e2808a-iptables-alerter-script\") pod \"iptables-alerter-8bqn2\" (UID: \"ef0b0f61-4619-4c40-afd3-0a83e4e2808a\") " pod="openshift-network-operator/iptables-alerter-8bqn2" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-sysconfig\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-run\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.974643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:53:56.975320 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974660 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/81e0c9ba-7535-45d5-933f-4aa8c33194c0-agent-certs\") pod \"konnectivity-agent-dmd4j\" (UID: \"81e0c9ba-7535-45d5-933f-4aa8c33194c0\") " pod="kube-system/konnectivity-agent-dmd4j" Sep 06 05:53:56.975320 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-host\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.975320 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974707 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce82693f-a593-41cf-83c0-d09f07d50168-host\") pod \"node-ca-cb5np\" (UID: \"ce82693f-a593-41cf-83c0-d09f07d50168\") " pod="openshift-image-registry/node-ca-cb5np" Sep 06 05:53:56.975320 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:53:56.975320 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcszm\" (UniqueName: \"kubernetes.io/projected/e5ada9e9-c4df-486d-9325-b5dcecdd451b-kube-api-access-rcszm\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:53:56.975320 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27b14080-015b-496f-b375-cc4cbdbf1892-hosts-file\") pod \"node-resolver-xxk4z\" (UID: \"27b14080-015b-496f-b375-cc4cbdbf1892\") " pod="openshift-dns/node-resolver-xxk4z" Sep 06 05:53:56.975320 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974935 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27b14080-015b-496f-b375-cc4cbdbf1892-hosts-file\") pod \"node-resolver-xxk4z\" (UID: \"27b14080-015b-496f-b375-cc4cbdbf1892\") " pod="openshift-dns/node-resolver-xxk4z" Sep 06 05:53:56.975320 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.974984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef0b0f61-4619-4c40-afd3-0a83e4e2808a-host-slash\") pod \"iptables-alerter-8bqn2\" (UID: \"ef0b0f61-4619-4c40-afd3-0a83e4e2808a\") " pod="openshift-network-operator/iptables-alerter-8bqn2" Sep 06 05:53:56.975320 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.975028 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-kubernetes\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.975320 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.975155 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-sysctl-d\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.975320 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.975274 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-sysctl-conf\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.975808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.975392 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e95c4ec-ee79-4651-9f84-6f1bd6660624-kubelet-config\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:53:56.975808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.975451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-run\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.975808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.975517 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-systemd\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.975808 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:56.975552 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:53:56.975808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.975565 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-sys\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.975808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.975599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-modprobe-d\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.975808 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:56.975621 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret podName:8e95c4ec-ee79-4651-9f84-6f1bd6660624 nodeName:}" failed. No retries permitted until 2026-09-06 05:53:57.475602651 +0000 UTC m=+3.124221829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret") pod "global-pull-secret-syncer-nksgl" (UID: "8e95c4ec-ee79-4651-9f84-6f1bd6660624") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:53:56.975808 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.975644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-lib-modules\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.976221 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.975842 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-etc-sysconfig\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.976221 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.975947 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-var-lib-kubelet\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.976221 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.976179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef0b0f61-4619-4c40-afd3-0a83e4e2808a-iptables-alerter-script\") pod \"iptables-alerter-8bqn2\" (UID: \"ef0b0f61-4619-4c40-afd3-0a83e4e2808a\") " pod="openshift-network-operator/iptables-alerter-8bqn2" Sep 06 05:53:56.976221 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.976198 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e95c4ec-ee79-4651-9f84-6f1bd6660624-dbus\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:53:56.976449 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.976423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/81e0c9ba-7535-45d5-933f-4aa8c33194c0-konnectivity-ca\") pod \"konnectivity-agent-dmd4j\" (UID: \"81e0c9ba-7535-45d5-933f-4aa8c33194c0\") " pod="kube-system/konnectivity-agent-dmd4j" Sep 06 05:53:56.976509 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.976476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b14080-015b-496f-b375-cc4cbdbf1892-tmp-dir\") pod \"node-resolver-xxk4z\" (UID: \"27b14080-015b-496f-b375-cc4cbdbf1892\") " pod="openshift-dns/node-resolver-xxk4z" Sep 06 05:53:56.976559 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:56.976521 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:53:56.976559 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.976550 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c80ff024-ecdd-468b-a518-167183800b69-host\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.976662 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:56.976567 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs podName:e5ada9e9-c4df-486d-9325-b5dcecdd451b nodeName:}" failed. No retries permitted until 2026-09-06 05:53:57.476550456 +0000 UTC m=+3.125169617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs") pod "network-metrics-daemon-2z6qh" (UID: "e5ada9e9-c4df-486d-9325-b5dcecdd451b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:53:56.976662 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.976600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce82693f-a593-41cf-83c0-d09f07d50168-host\") pod \"node-ca-cb5np\" (UID: \"ce82693f-a593-41cf-83c0-d09f07d50168\") " pod="openshift-image-registry/node-ca-cb5np" Sep 06 05:53:56.977175 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.977152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce82693f-a593-41cf-83c0-d09f07d50168-serviceca\") pod \"node-ca-cb5np\" (UID: \"ce82693f-a593-41cf-83c0-d09f07d50168\") " pod="openshift-image-registry/node-ca-cb5np" Sep 06 05:53:56.978635 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.978612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/81e0c9ba-7535-45d5-933f-4aa8c33194c0-agent-certs\") pod \"konnectivity-agent-dmd4j\" (UID: \"81e0c9ba-7535-45d5-933f-4aa8c33194c0\") " pod="kube-system/konnectivity-agent-dmd4j" Sep 06 05:53:56.979419 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.979401 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c80ff024-ecdd-468b-a518-167183800b69-tmp\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.981303 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.981282 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c80ff024-ecdd-468b-a518-167183800b69-etc-tuned\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.988552 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.988530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sgj8\" (UniqueName: \"kubernetes.io/projected/c80ff024-ecdd-468b-a518-167183800b69-kube-api-access-2sgj8\") pod \"tuned-vrsf2\" (UID: \"c80ff024-ecdd-468b-a518-167183800b69\") " pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:56.989443 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.989412 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbqc4\" (UniqueName: \"kubernetes.io/projected/ef0b0f61-4619-4c40-afd3-0a83e4e2808a-kube-api-access-tbqc4\") pod \"iptables-alerter-8bqn2\" (UID: \"ef0b0f61-4619-4c40-afd3-0a83e4e2808a\") " pod="openshift-network-operator/iptables-alerter-8bqn2" Sep 06 05:53:56.989606 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.989586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjjv\" (UniqueName: \"kubernetes.io/projected/27b14080-015b-496f-b375-cc4cbdbf1892-kube-api-access-7kjjv\") pod \"node-resolver-xxk4z\" (UID: \"27b14080-015b-496f-b375-cc4cbdbf1892\") " pod="openshift-dns/node-resolver-xxk4z" Sep 06 05:53:56.990425 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.990405 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcszm\" (UniqueName: \"kubernetes.io/projected/e5ada9e9-c4df-486d-9325-b5dcecdd451b-kube-api-access-rcszm\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:53:56.990937 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:56.990915 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjgtc\" (UniqueName: \"kubernetes.io/projected/ce82693f-a593-41cf-83c0-d09f07d50168-kube-api-access-tjgtc\") pod \"node-ca-cb5np\" (UID: \"ce82693f-a593-41cf-83c0-d09f07d50168\") " pod="openshift-image-registry/node-ca-cb5np" Sep 06 05:53:57.060256 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.060232 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cqfs6" Sep 06 05:53:57.065978 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.065954 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gjhdl" Sep 06 05:53:57.074973 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.074789 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:53:57.075793 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:57.075760 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678e8aa3_c088_4f39_bf3d_f158f45c9aef.slice/crio-8f7a3b27fdfa96fdb50d2110740c78856be6909ea6cec36ed45a80d4491d6193 WatchSource:0}: Error finding container 8f7a3b27fdfa96fdb50d2110740c78856be6909ea6cec36ed45a80d4491d6193: Status 404 returned error can't find the container with id 8f7a3b27fdfa96fdb50d2110740c78856be6909ea6cec36ed45a80d4491d6193 Sep 06 05:53:57.080161 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.079354 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" Sep 06 05:53:57.085161 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.085140 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xxk4z" Sep 06 05:53:57.089499 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:57.089436 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffcd62a_9c1d_45fe_a3eb_1e5e040b20d9.slice/crio-9ba2d78059244429d93c1213ef4e8077db5e17bd07e109449be06e8c475fc274 WatchSource:0}: Error finding container 9ba2d78059244429d93c1213ef4e8077db5e17bd07e109449be06e8c475fc274: Status 404 returned error can't find the container with id 9ba2d78059244429d93c1213ef4e8077db5e17bd07e109449be06e8c475fc274 Sep 06 05:53:57.090677 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.090661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cb5np" Sep 06 05:53:57.092518 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:57.092489 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08082397_d99f_4ed1_af18_c48acdeb20e0.slice/crio-f947e619068459363bbdcde2a5c8c9c07ff872903aee8d9c1c5dbe183839f77a WatchSource:0}: Error finding container f947e619068459363bbdcde2a5c8c9c07ff872903aee8d9c1c5dbe183839f77a: Status 404 returned error can't find the container with id f947e619068459363bbdcde2a5c8c9c07ff872903aee8d9c1c5dbe183839f77a Sep 06 05:53:57.095156 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.095138 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dmd4j" Sep 06 05:53:57.101036 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.101017 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8bqn2" Sep 06 05:53:57.104461 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.104443 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" Sep 06 05:53:57.108333 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:57.108103 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce82693f_a593_41cf_83c0_d09f07d50168.slice/crio-00104fa4f0f1611f6469a95cd20476b97e730b8f25f17959ab57615a08f694d8 WatchSource:0}: Error finding container 00104fa4f0f1611f6469a95cd20476b97e730b8f25f17959ab57615a08f694d8: Status 404 returned error can't find the container with id 00104fa4f0f1611f6469a95cd20476b97e730b8f25f17959ab57615a08f694d8 Sep 06 05:53:57.110349 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:57.110299 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e0c9ba_7535_45d5_933f_4aa8c33194c0.slice/crio-9558f3f2d108a4c6c68c7273b5a23062b47c04f5314b40d0bbed4cb14ed422fd WatchSource:0}: Error finding container 9558f3f2d108a4c6c68c7273b5a23062b47c04f5314b40d0bbed4cb14ed422fd: Status 404 returned error can't find the container with id 9558f3f2d108a4c6c68c7273b5a23062b47c04f5314b40d0bbed4cb14ed422fd Sep 06 05:53:57.114286 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:57.114266 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0b0f61_4619_4c40_afd3_0a83e4e2808a.slice/crio-ac2f4277dfdee602beff959308954f4ae9a9f2e66a863fe645911a74c66b363d WatchSource:0}: Error finding container ac2f4277dfdee602beff959308954f4ae9a9f2e66a863fe645911a74c66b363d: Status 404 returned error can't find the container with id ac2f4277dfdee602beff959308954f4ae9a9f2e66a863fe645911a74c66b363d Sep 06 05:53:57.120500 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:53:57.120480 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80ff024_ecdd_468b_a518_167183800b69.slice/crio-4811aecdf99c444413bcc3c69475d19dfd21969288cefe69e0485a338dfb4fec WatchSource:0}: Error finding container 4811aecdf99c444413bcc3c69475d19dfd21969288cefe69e0485a338dfb4fec: Status 404 returned error can't find the container with id 4811aecdf99c444413bcc3c69475d19dfd21969288cefe69e0485a338dfb4fec Sep 06 05:53:57.378462 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.378391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns46\" (UniqueName: \"kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46\") pod \"network-check-target-jlt55\" (UID: \"32d81600-7fda-434c-8df0-e43a6c3bd84c\") " pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:53:57.378612 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:57.378540 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:53:57.378612 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:57.378562 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:53:57.378612 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:57.378574 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gns46 for pod openshift-network-diagnostics/network-check-target-jlt55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:53:57.378754 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:57.378642 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46 podName:32d81600-7fda-434c-8df0-e43a6c3bd84c nodeName:}" failed. No retries permitted until 2026-09-06 05:53:58.378622727 +0000 UTC m=+4.027241908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gns46" (UniqueName: "kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46") pod "network-check-target-jlt55" (UID: "32d81600-7fda-434c-8df0-e43a6c3bd84c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:53:57.479373 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.479334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:53:57.479553 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.479392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:53:57.479553 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:57.479498 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:53:57.479553 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:57.479500 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:53:57.479553 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:57.479554 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs podName:e5ada9e9-c4df-486d-9325-b5dcecdd451b nodeName:}" failed. No retries permitted until 2026-09-06 05:53:58.479535807 +0000 UTC m=+4.128154965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs") pod "network-metrics-daemon-2z6qh" (UID: "e5ada9e9-c4df-486d-9325-b5dcecdd451b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:53:57.479714 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:57.479596 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret podName:8e95c4ec-ee79-4651-9f84-6f1bd6660624 nodeName:}" failed. No retries permitted until 2026-09-06 05:53:58.479586677 +0000 UTC m=+4.128205834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret") pod "global-pull-secret-syncer-nksgl" (UID: "8e95c4ec-ee79-4651-9f84-6f1bd6660624") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:53:57.811604 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.811525 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 05:48:56 +0000 UTC" deadline="2028-06-14 17:04:02.182682886 +0000 UTC" Sep 06 05:53:57.811604 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.811555 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15539h10m4.371131664s" Sep 06 05:53:57.879859 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.879797 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqfs6" event={"ID":"678e8aa3-c088-4f39-bf3d-f158f45c9aef","Type":"ContainerStarted","Data":"8f7a3b27fdfa96fdb50d2110740c78856be6909ea6cec36ed45a80d4491d6193"} Sep 06 05:53:57.881124 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.881090 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dmd4j" event={"ID":"81e0c9ba-7535-45d5-933f-4aa8c33194c0","Type":"ContainerStarted","Data":"9558f3f2d108a4c6c68c7273b5a23062b47c04f5314b40d0bbed4cb14ed422fd"} Sep 06 05:53:57.882278 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.882257 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cb5np" event={"ID":"ce82693f-a593-41cf-83c0-d09f07d50168","Type":"ContainerStarted","Data":"00104fa4f0f1611f6469a95cd20476b97e730b8f25f17959ab57615a08f694d8"} Sep 06 05:53:57.883325 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.883308 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" event={"ID":"c80ff024-ecdd-468b-a518-167183800b69","Type":"ContainerStarted","Data":"4811aecdf99c444413bcc3c69475d19dfd21969288cefe69e0485a338dfb4fec"} Sep 06 05:53:57.884372 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.884356 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8bqn2" event={"ID":"ef0b0f61-4619-4c40-afd3-0a83e4e2808a","Type":"ContainerStarted","Data":"ac2f4277dfdee602beff959308954f4ae9a9f2e66a863fe645911a74c66b363d"} Sep 06 05:53:57.885526 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.885503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xxk4z" event={"ID":"27b14080-015b-496f-b375-cc4cbdbf1892","Type":"ContainerStarted","Data":"868f91e0bac6015c305c335d227fe74bb57bc53c03baf5528492e180e006ff8e"} Sep 06 05:53:57.886777 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.886758 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" event={"ID":"08082397-d99f-4ed1-af18-c48acdeb20e0","Type":"ContainerStarted","Data":"f947e619068459363bbdcde2a5c8c9c07ff872903aee8d9c1c5dbe183839f77a"} Sep 06 05:53:57.888044 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.888020 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" event={"ID":"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9","Type":"ContainerStarted","Data":"9ba2d78059244429d93c1213ef4e8077db5e17bd07e109449be06e8c475fc274"} Sep 06 05:53:57.889257 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:57.889226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gjhdl" event={"ID":"513bdbba-c5b6-4ea9-80ea-f909cbba8abb","Type":"ContainerStarted","Data":"6b3fc49c4f71971fa7f26b52c43486f227d44289b0f9aee25aa1780a3f8967d5"} Sep 06 05:53:58.385935 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:58.385901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns46\" (UniqueName: \"kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46\") pod \"network-check-target-jlt55\" (UID: \"32d81600-7fda-434c-8df0-e43a6c3bd84c\") " pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:53:58.386120 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:58.386038 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:53:58.386120 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:58.386055 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:53:58.386120 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:58.386063 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gns46 for pod openshift-network-diagnostics/network-check-target-jlt55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:53:58.386120 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:58.386108 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46 podName:32d81600-7fda-434c-8df0-e43a6c3bd84c nodeName:}" failed. No retries permitted until 2026-09-06 05:54:00.3860959 +0000 UTC m=+6.034715058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gns46" (UniqueName: "kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46") pod "network-check-target-jlt55" (UID: "32d81600-7fda-434c-8df0-e43a6c3bd84c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:53:58.486716 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:58.486682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:53:58.486898 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:58.486733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:53:58.486898 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:58.486870 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:53:58.487016 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:58.486933 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs podName:e5ada9e9-c4df-486d-9325-b5dcecdd451b nodeName:}" failed. No retries permitted until 2026-09-06 05:54:00.486913774 +0000 UTC m=+6.135532956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs") pod "network-metrics-daemon-2z6qh" (UID: "e5ada9e9-c4df-486d-9325-b5dcecdd451b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:53:58.487358 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:58.487294 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:53:58.490914 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:58.490893 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret podName:8e95c4ec-ee79-4651-9f84-6f1bd6660624 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:00.490822168 +0000 UTC m=+6.139441327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret") pod "global-pull-secret-syncer-nksgl" (UID: "8e95c4ec-ee79-4651-9f84-6f1bd6660624") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:53:58.811979 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:58.811910 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 05:48:56 +0000 UTC" deadline="2028-04-24 07:00:49.539611854 +0000 UTC" Sep 06 05:53:58.811979 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:58.811965 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14305h6m50.727651097s" Sep 06 05:53:58.870636 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:58.870589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:53:58.870780 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:58.870735 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:53:58.872110 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:58.871436 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:53:58.872110 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:58.871531 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:53:58.872110 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:58.871609 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:53:58.872110 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:53:58.871674 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:53:58.900851 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:58.900763 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-34.ec2.internal" event={"ID":"6b5baafa68872d0e1a89c3b7fe397e3a","Type":"ContainerStarted","Data":"c6ada315df83fa392f35cd9b564aee74d230940b5e8e4b5f1c024dd388b13bbb"} Sep 06 05:53:59.919076 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:59.919034 2580 generic.go:356] "Generic (PLEG): container finished" podID="e13a78121acfe93071c1c9eb406a02ec" containerID="740a29d022e8554a288d8a1b466ab3078bbeea78e02ed083d2e684474e98b4fc" exitCode=0 Sep 06 05:53:59.920005 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:59.919979 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" event={"ID":"e13a78121acfe93071c1c9eb406a02ec","Type":"ContainerDied","Data":"740a29d022e8554a288d8a1b466ab3078bbeea78e02ed083d2e684474e98b4fc"} Sep 06 05:53:59.936389 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:53:59.936340 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-34.ec2.internal" podStartSLOduration=4.9363223529999996 podStartE2EDuration="4.936322353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:53:58.915928678 +0000 UTC m=+4.564547859" watchObservedRunningTime="2026-09-06 05:53:59.936322353 +0000 UTC m=+5.584941533" Sep 06 05:54:00.403722 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:00.403646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns46\" (UniqueName: \"kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46\") pod \"network-check-target-jlt55\" (UID: \"32d81600-7fda-434c-8df0-e43a6c3bd84c\") " pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:00.403901 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:00.403882 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:54:00.403970 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:00.403906 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:54:00.403970 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:00.403920 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gns46 for pod openshift-network-diagnostics/network-check-target-jlt55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:00.404076 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:00.403990 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46 podName:32d81600-7fda-434c-8df0-e43a6c3bd84c nodeName:}" failed. No retries permitted until 2026-09-06 05:54:04.403964202 +0000 UTC m=+10.052583376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gns46" (UniqueName: "kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46") pod "network-check-target-jlt55" (UID: "32d81600-7fda-434c-8df0-e43a6c3bd84c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:00.504803 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:00.504766 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:00.505003 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:00.504880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:00.505070 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:00.505010 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:00.505070 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:00.505069 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret podName:8e95c4ec-ee79-4651-9f84-6f1bd6660624 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:04.505053067 +0000 UTC m=+10.153672236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret") pod "global-pull-secret-syncer-nksgl" (UID: "8e95c4ec-ee79-4651-9f84-6f1bd6660624") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:00.505466 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:00.505444 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:00.505466 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:00.505498 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs podName:e5ada9e9-c4df-486d-9325-b5dcecdd451b nodeName:}" failed. No retries permitted until 2026-09-06 05:54:04.505482888 +0000 UTC m=+10.154102048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs") pod "network-metrics-daemon-2z6qh" (UID: "e5ada9e9-c4df-486d-9325-b5dcecdd451b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:00.868583 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:00.868508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:00.868759 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:00.868643 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:00.868759 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:00.868745 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:00.868932 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:00.868638 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:00.868932 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:00.868797 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:00.868932 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:00.868874 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:00.923351 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:00.923322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" event={"ID":"e13a78121acfe93071c1c9eb406a02ec","Type":"ContainerStarted","Data":"846e24ad17e32bb7e64aa26d40d3fa8e71da757efe7edc56c904c71e09374d51"} Sep 06 05:54:02.871696 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:02.871505 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:02.872139 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:02.871577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:02.872139 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:02.871787 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:02.872139 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:02.871589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:02.872139 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:02.871910 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:02.872139 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:02.871985 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:04.438148 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:04.438106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns46\" (UniqueName: \"kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46\") pod \"network-check-target-jlt55\" (UID: \"32d81600-7fda-434c-8df0-e43a6c3bd84c\") " pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:04.438621 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:04.438320 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:54:04.438621 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:04.438350 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:54:04.438621 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:04.438364 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gns46 for pod openshift-network-diagnostics/network-check-target-jlt55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:04.438621 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:04.438435 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46 podName:32d81600-7fda-434c-8df0-e43a6c3bd84c nodeName:}" failed. No retries permitted until 2026-09-06 05:54:12.438415027 +0000 UTC m=+18.087034207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gns46" (UniqueName: "kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46") pod "network-check-target-jlt55" (UID: "32d81600-7fda-434c-8df0-e43a6c3bd84c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:04.539423 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:04.539339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:04.539423 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:04.539392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:04.539650 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:04.539506 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:04.539650 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:04.539571 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs podName:e5ada9e9-c4df-486d-9325-b5dcecdd451b nodeName:}" failed. No retries permitted until 2026-09-06 05:54:12.539552274 +0000 UTC m=+18.188171447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs") pod "network-metrics-daemon-2z6qh" (UID: "e5ada9e9-c4df-486d-9325-b5dcecdd451b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:04.539650 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:04.539508 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:04.539650 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:04.539641 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret podName:8e95c4ec-ee79-4651-9f84-6f1bd6660624 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:12.539626835 +0000 UTC m=+18.188246004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret") pod "global-pull-secret-syncer-nksgl" (UID: "8e95c4ec-ee79-4651-9f84-6f1bd6660624") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:04.869728 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:04.869612 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:04.869728 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:04.869719 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:04.870060 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:04.870027 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:04.870194 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:04.870168 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:04.870266 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:04.870210 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:04.870316 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:04.870294 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:06.869149 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:06.869093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:06.869149 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:06.869114 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:06.869655 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:06.869213 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:06.869655 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:06.869269 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:06.869655 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:06.869431 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:06.869655 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:06.869489 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:08.872221 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:08.872190 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:08.872221 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:08.872210 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:08.872221 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:08.872217 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:08.872726 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:08.872300 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:08.872726 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:08.872380 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:08.872726 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:08.872487 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:10.868607 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:10.868567 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:10.868607 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:10.868589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:10.868607 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:10.868583 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:10.869227 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:10.868702 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:10.869227 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:10.868742 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:10.869227 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:10.868854 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:12.495818 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:12.495780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns46\" (UniqueName: \"kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46\") pod \"network-check-target-jlt55\" (UID: \"32d81600-7fda-434c-8df0-e43a6c3bd84c\") " pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:12.496282 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:12.495958 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:54:12.496282 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:12.495983 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:54:12.496282 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:12.495994 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gns46 for pod openshift-network-diagnostics/network-check-target-jlt55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:12.496282 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:12.496051 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46 podName:32d81600-7fda-434c-8df0-e43a6c3bd84c nodeName:}" failed. No retries permitted until 2026-09-06 05:54:28.49603489 +0000 UTC m=+34.144654067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gns46" (UniqueName: "kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46") pod "network-check-target-jlt55" (UID: "32d81600-7fda-434c-8df0-e43a6c3bd84c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:12.596988 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:12.596949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:12.596988 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:12.597000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:12.597205 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:12.597128 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:12.597205 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:12.597184 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs podName:e5ada9e9-c4df-486d-9325-b5dcecdd451b nodeName:}" failed. No retries permitted until 2026-09-06 05:54:28.597169241 +0000 UTC m=+34.245788414 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs") pod "network-metrics-daemon-2z6qh" (UID: "e5ada9e9-c4df-486d-9325-b5dcecdd451b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:12.597317 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:12.597125 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:12.597317 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:12.597274 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret podName:8e95c4ec-ee79-4651-9f84-6f1bd6660624 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:28.597254288 +0000 UTC m=+34.245873449 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret") pod "global-pull-secret-syncer-nksgl" (UID: "8e95c4ec-ee79-4651-9f84-6f1bd6660624") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:12.869167 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:12.869081 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:12.869167 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:12.869115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:12.869167 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:12.869081 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:12.869444 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:12.869222 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:12.869444 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:12.869307 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:12.869444 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:12.869392 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:14.869968 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:14.869931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:14.870424 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:14.870042 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:14.870424 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:14.870049 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:14.870424 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:14.870083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:14.870424 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:14.870173 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:14.870424 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:14.870277 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:15.952322 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.952290 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" event={"ID":"c80ff024-ecdd-468b-a518-167183800b69","Type":"ContainerStarted","Data":"591ceda8c4c3ff1bbb5f2bca22798d5a69024bde249af8af39d32ae80416baac"} Sep 06 05:54:15.953864 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.953823 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xxk4z" event={"ID":"27b14080-015b-496f-b375-cc4cbdbf1892","Type":"ContainerStarted","Data":"2197c068ea4906eed364017aaabf1bff7f14d3207cd66fa8f2f7f8a05694e509"} Sep 06 05:54:15.955264 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.955243 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" event={"ID":"08082397-d99f-4ed1-af18-c48acdeb20e0","Type":"ContainerStarted","Data":"909c87589a2c6ea517a5aa4c7491d949c4bf7b70e8e87eb152c62889b2c148ee"} Sep 06 05:54:15.959042 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.959017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" event={"ID":"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9","Type":"ContainerStarted","Data":"3899539a80761745b460a515da887838cfbd59740573396874d2ba6d92c970e7"} Sep 06 05:54:15.959119 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.959100 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" event={"ID":"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9","Type":"ContainerStarted","Data":"45797f4e9bc45991e3fc6eb0f10bfc3dd62d830656d9c7e2cf5ee7a2870aae67"} Sep 06 05:54:15.959173 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.959162 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" event={"ID":"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9","Type":"ContainerStarted","Data":"d8325562790dcb51cbaba6c96412f45cae6f47d59443dda55cdee4063b48ba98"} Sep 06 05:54:15.959223 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.959207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" event={"ID":"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9","Type":"ContainerStarted","Data":"3276ae99d230f73a1a0b8ececf49cfba9cfeb9aa1a1b1d76005aed8f175f80a3"} Sep 06 05:54:15.959270 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.959249 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" event={"ID":"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9","Type":"ContainerStarted","Data":"08f8fcd94bc036621dfd0b72dd518201a1e03ddcbf21c78d177b499194dd2d91"} Sep 06 05:54:15.960406 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.960385 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gjhdl" event={"ID":"513bdbba-c5b6-4ea9-80ea-f909cbba8abb","Type":"ContainerStarted","Data":"ee1f3cc621bb09ebb4d1d0f090064f8dd5c037b9b8c4561f1908cc73fee480a0"} Sep 06 05:54:15.961785 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.961763 2580 generic.go:356] "Generic (PLEG): container finished" podID="678e8aa3-c088-4f39-bf3d-f158f45c9aef" containerID="f0f5a4b8db8ce4978f9311f9082630d5e390b81069ec91a35b5292eb76f9a3be" exitCode=0 Sep 06 05:54:15.961899 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.961847 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqfs6" event={"ID":"678e8aa3-c088-4f39-bf3d-f158f45c9aef","Type":"ContainerDied","Data":"f0f5a4b8db8ce4978f9311f9082630d5e390b81069ec91a35b5292eb76f9a3be"} Sep 06 05:54:15.963246 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.963227 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dmd4j" event={"ID":"81e0c9ba-7535-45d5-933f-4aa8c33194c0","Type":"ContainerStarted","Data":"520c0b428149ca2dd4354645de8bc34135539e557eeeef2978256984838a6bf3"} Sep 06 05:54:15.964620 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.964603 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cb5np" event={"ID":"ce82693f-a593-41cf-83c0-d09f07d50168","Type":"ContainerStarted","Data":"2e0b4b3b615032cbbf8e3933e6e83c39ef5c489ee54d3ef60c2b24421a1f05b7"} Sep 06 05:54:15.967385 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.967347 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-34.ec2.internal" podStartSLOduration=20.967335515 podStartE2EDuration="20.967335515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:00.946075325 +0000 UTC m=+6.594694504" watchObservedRunningTime="2026-09-06 05:54:15.967335515 +0000 UTC m=+21.615954696" Sep 06 05:54:15.985674 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:15.985634 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vrsf2" podStartSLOduration=2.758106818 podStartE2EDuration="20.985620853s" totalImagesPullingTime="18.227514035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:15.967269452 +0000 UTC m=+21.615888632" watchObservedRunningTime="2026-09-06 05:54:15.985620853 +0000 UTC m=+21.634240032" Sep 06 05:54:16.026861 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.026756 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xxk4z" podStartSLOduration=6.803233063 podStartE2EDuration="21.026741513s" totalImagesPullingTime="14.22350845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:16.004386158 +0000 UTC m=+21.653005337" watchObservedRunningTime="2026-09-06 05:54:16.026741513 +0000 UTC m=+21.675360744" Sep 06 05:54:16.026986 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.026905 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gjhdl" podStartSLOduration=3.718348469 podStartE2EDuration="22.02689917s" totalImagesPullingTime="18.308550701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:16.026422021 +0000 UTC m=+21.675041199" watchObservedRunningTime="2026-09-06 05:54:16.02689917 +0000 UTC m=+21.675518348" Sep 06 05:54:16.053265 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.053216 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cb5np" podStartSLOduration=2.825650227 podStartE2EDuration="21.053199842s" totalImagesPullingTime="18.227549615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:16.039513854 +0000 UTC m=+21.688133044" watchObservedRunningTime="2026-09-06 05:54:16.053199842 +0000 UTC m=+21.701819020" Sep 06 05:54:16.053507 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.053487 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dmd4j" podStartSLOduration=2.817251012 podStartE2EDuration="21.053483329s" totalImagesPullingTime="18.236232317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:16.052996258 +0000 UTC m=+21.701615448" watchObservedRunningTime="2026-09-06 05:54:16.053483329 +0000 UTC m=+21.702102508" Sep 06 05:54:16.575094 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.575066 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 06 05:54:16.828394 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.828297 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-06T05:54:16.575089221Z","UUID":"b15a3b51-ce61-4e95-b23a-8bb1428d90bc","Handler":null,"Name":"","Endpoint":""} Sep 06 05:54:16.830164 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.830143 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 06 05:54:16.830297 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.830171 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 06 05:54:16.869258 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.869238 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:16.869389 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.869271 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:16.869389 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.869339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:16.869389 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:16.869350 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:16.869546 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:16.869416 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:16.869546 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:16.869506 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:16.967821 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.967785 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8bqn2" event={"ID":"ef0b0f61-4619-4c40-afd3-0a83e4e2808a","Type":"ContainerStarted","Data":"1acbfcd7807803cd2f2ec3b77669f7e4b65a246716281c5be261b87ffe45fcf3"} Sep 06 05:54:16.969620 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.969599 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" event={"ID":"08082397-d99f-4ed1-af18-c48acdeb20e0","Type":"ContainerStarted","Data":"9d82f863123fe86246e2e84bc9e1d4a74db1e4a6cc716f04a1fe0b4f27206bd3"} Sep 06 05:54:16.972671 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.972652 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" event={"ID":"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9","Type":"ContainerStarted","Data":"54a399dde080f5d699938e70e433877690ba6804b18fb22f6daa21a5e1efad07"} Sep 06 05:54:16.981876 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:16.981820 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8bqn2" podStartSLOduration=3.749748207 podStartE2EDuration="21.981808389s" totalImagesPullingTime="18.232060182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:16.981362209 +0000 UTC m=+22.629981387" watchObservedRunningTime="2026-09-06 05:54:16.981808389 +0000 UTC m=+22.630427569" Sep 06 05:54:17.977206 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:17.977165 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" event={"ID":"08082397-d99f-4ed1-af18-c48acdeb20e0","Type":"ContainerStarted","Data":"8bc98b08cda68a2eef32051d97acfb228cc8e5e7e7443687961b9e6d44ef260c"} Sep 06 05:54:17.993508 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:17.993454 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvhv2" podStartSLOduration=3.57117395 podStartE2EDuration="23.993438155s" totalImagesPullingTime="20.422264205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:17.993416468 +0000 UTC m=+23.642035647" watchObservedRunningTime="2026-09-06 05:54:17.993438155 +0000 UTC m=+23.642057335" Sep 06 05:54:18.868956 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:18.868915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:18.869217 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:18.869041 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:18.869217 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:18.868918 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:18.869217 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:18.869117 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:18.869217 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:18.868915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:18.869217 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:18.869176 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:18.983921 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:18.983881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" event={"ID":"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9","Type":"ContainerStarted","Data":"65aeb7827343bb8cc01922ec8084afb6070c46044c7877a5d42f4168d390f888"} Sep 06 05:54:19.468981 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:19.468945 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dmd4j" Sep 06 05:54:19.469589 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:19.469569 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dmd4j" Sep 06 05:54:19.986864 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:19.986813 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dmd4j" Sep 06 05:54:19.987397 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:19.987379 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dmd4j" Sep 06 05:54:20.868728 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:20.868564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:20.868857 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:20.868564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:20.868857 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:20.868603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:20.868965 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:20.868845 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:20.868965 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:20.868898 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:20.869057 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:20.868965 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:20.992543 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:20.992511 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" event={"ID":"dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9","Type":"ContainerStarted","Data":"94a2b34af88a32613d05b2fda5f6ee2288c501cbdfaaf09897937ab72db30edc"} Sep 06 05:54:20.993153 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:20.992727 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:54:20.994461 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:20.994435 2580 generic.go:356] "Generic (PLEG): container finished" podID="678e8aa3-c088-4f39-bf3d-f158f45c9aef" containerID="94736f8512d962fcc21619a1e11a55d171e10eb9bf23bf9423ba7091819bf5b8" exitCode=0 Sep 06 05:54:20.994554 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:20.994513 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqfs6" event={"ID":"678e8aa3-c088-4f39-bf3d-f158f45c9aef","Type":"ContainerDied","Data":"94736f8512d962fcc21619a1e11a55d171e10eb9bf23bf9423ba7091819bf5b8"} Sep 06 05:54:21.008147 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:21.008123 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:54:21.017931 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:21.017891 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" podStartSLOduration=7.579256623 podStartE2EDuration="26.017878616s" totalImagesPullingTime="18.438621993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:21.016350776 +0000 UTC m=+26.664969954" watchObservedRunningTime="2026-09-06 05:54:21.017878616 +0000 UTC m=+26.666497794" Sep 06 05:54:21.997537 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:21.997464 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 06 05:54:21.998192 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:21.997639 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:54:22.017194 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:22.017171 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:54:22.514166 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:22.513964 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2z6qh"] Sep 06 05:54:22.514296 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:22.514263 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:22.514376 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:22.514359 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:22.516382 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:22.516360 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nksgl"] Sep 06 05:54:22.516494 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:22.516457 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:22.516574 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:22.516551 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:22.528118 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:22.528093 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jlt55"] Sep 06 05:54:22.528212 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:22.528186 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:22.528293 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:22.528273 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:23.001246 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:23.001212 2580 generic.go:356] "Generic (PLEG): container finished" podID="678e8aa3-c088-4f39-bf3d-f158f45c9aef" containerID="00872add4fa2bae43ecf49eee123c7cf90c8fb3403a06ee6c5c31edf9ac73bcb" exitCode=0 Sep 06 05:54:23.001650 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:23.001296 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqfs6" event={"ID":"678e8aa3-c088-4f39-bf3d-f158f45c9aef","Type":"ContainerDied","Data":"00872add4fa2bae43ecf49eee123c7cf90c8fb3403a06ee6c5c31edf9ac73bcb"} Sep 06 05:54:23.001650 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:23.001344 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 06 05:54:23.868681 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:23.868653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:23.868779 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:23.868680 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:23.868779 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:23.868684 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:23.868852 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:23.868805 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:23.868934 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:23.868907 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:23.868981 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:23.868967 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:24.005430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:24.005396 2580 generic.go:356] "Generic (PLEG): container finished" podID="678e8aa3-c088-4f39-bf3d-f158f45c9aef" containerID="ce0ad693d6aa9da307949d87ec0cbaa78de848730e8c9869b332e6a232ee5364" exitCode=0 Sep 06 05:54:24.005782 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:24.005476 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqfs6" event={"ID":"678e8aa3-c088-4f39-bf3d-f158f45c9aef","Type":"ContainerDied","Data":"ce0ad693d6aa9da307949d87ec0cbaa78de848730e8c9869b332e6a232ee5364"} Sep 06 05:54:24.005782 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:24.005564 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 06 05:54:24.567707 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:24.567670 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:54:25.019408 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:25.019312 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" podUID="dffcd62a-9c1d-45fe-a3eb-1e5e040b20d9" containerName="ovnkube-controller" probeResult="failure" output="" Sep 06 05:54:25.868882 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:25.868852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:25.868882 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:25.868872 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:25.869118 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:25.868978 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:25.869118 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:25.869017 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:25.869214 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:25.869116 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:25.869214 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:25.869192 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:27.869310 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:27.869275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:27.869801 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:27.869285 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:27.869801 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:27.869391 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:54:27.869801 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:27.869477 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlt55" podUID="32d81600-7fda-434c-8df0-e43a6c3bd84c" Sep 06 05:54:27.869801 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:27.869291 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:27.869801 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:27.869574 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nksgl" podUID="8e95c4ec-ee79-4651-9f84-6f1bd6660624" Sep 06 05:54:28.168778 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.168532 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-34.ec2.internal" event="NodeReady" Sep 06 05:54:28.168966 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.168933 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 06 05:54:28.201058 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.201024 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-854f5cf89c-xtljb"] Sep 06 05:54:28.231463 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.231431 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-854f5cf89c-xtljb"] Sep 06 05:54:28.231639 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.231483 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5g9s6"] Sep 06 05:54:28.231639 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.231577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.234430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.234387 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qcp4t\"" Sep 06 05:54:28.234713 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.234691 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 06 05:54:28.234817 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.234697 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 06 05:54:28.234817 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.234697 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 06 05:54:28.241519 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.241497 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 06 05:54:28.247620 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.247599 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-f7dzz"] Sep 06 05:54:28.247756 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.247737 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.250883 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.250855 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 06 05:54:28.251002 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.250939 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ppscw\"" Sep 06 05:54:28.251002 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.250964 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 06 05:54:28.271061 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.271027 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x9r8b"] Sep 06 05:54:28.271215 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.271163 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:54:28.273720 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.273699 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 06 05:54:28.273838 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.273699 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 06 05:54:28.273838 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.273705 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nr686\"" Sep 06 05:54:28.294953 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.294915 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9r8b"] Sep 06 05:54:28.294953 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.294954 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-f7dzz"] Sep 06 05:54:28.295132 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.294970 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5g9s6"] Sep 06 05:54:28.295132 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.295023 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9r8b" Sep 06 05:54:28.297728 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.297702 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 06 05:54:28.297728 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.297718 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 06 05:54:28.297943 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.297781 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hqtmv\"" Sep 06 05:54:28.297943 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.297848 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 06 05:54:28.310050 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.310025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecd3697-4253-4467-a57b-368c0950ffe2-config-volume\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.310241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.310075 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-certificates\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.310241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.310124 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.310241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.310154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.310241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.310180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d264052-c7af-404d-9f73-3a5bed5d755d-ca-trust-extracted\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.310241 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.310208 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d264052-c7af-404d-9f73-3a5bed5d755d-installation-pull-secrets\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.310415 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.310264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmwdd\" (UniqueName: \"kubernetes.io/projected/1ecd3697-4253-4467-a57b-368c0950ffe2-kube-api-access-gmwdd\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.310415 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.310302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ecd3697-4253-4467-a57b-368c0950ffe2-tmp-dir\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.310415 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.310320 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbn2m\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-kube-api-access-xbn2m\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.310415 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.310339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d264052-c7af-404d-9f73-3a5bed5d755d-trusted-ca\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.310415 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.310354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-bound-sa-token\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.310415 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.310371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d264052-c7af-404d-9f73-3a5bed5d755d-image-registry-private-configuration\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.411334 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwdd\" (UniqueName: \"kubernetes.io/projected/1ecd3697-4253-4467-a57b-368c0950ffe2-kube-api-access-gmwdd\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.411506 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ecd3697-4253-4467-a57b-368c0950ffe2-tmp-dir\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.411506 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbn2m\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-kube-api-access-xbn2m\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.411506 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411417 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d264052-c7af-404d-9f73-3a5bed5d755d-trusted-ca\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.411664 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-bound-sa-token\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.411664 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411607 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d264052-c7af-404d-9f73-3a5bed5d755d-image-registry-private-configuration\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.411664 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4354b120-a2f7-43f8-b3e7-be55e37d543f-cert\") pod \"ingress-canary-x9r8b\" (UID: \"4354b120-a2f7-43f8-b3e7-be55e37d543f\") " pod="openshift-ingress-canary/ingress-canary-x9r8b" Sep 06 05:54:28.411664 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecd3697-4253-4467-a57b-368c0950ffe2-config-volume\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.411890 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411811 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ecd3697-4253-4467-a57b-368c0950ffe2-tmp-dir\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.411890 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-certificates\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.411996 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.411996 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411940 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:54:28.411996 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.411968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2s6w\" (UniqueName: \"kubernetes.io/projected/4354b120-a2f7-43f8-b3e7-be55e37d543f-kube-api-access-s2s6w\") pod \"ingress-canary-x9r8b\" (UID: \"4354b120-a2f7-43f8-b3e7-be55e37d543f\") " pod="openshift-ingress-canary/ingress-canary-x9r8b" Sep 06 05:54:28.412271 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.412020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.412271 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.412074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d264052-c7af-404d-9f73-3a5bed5d755d-ca-trust-extracted\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.412271 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.412077 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:54:28.412271 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.412117 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-854f5cf89c-xtljb: secret "image-registry-tls" not found Sep 06 05:54:28.412271 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.412159 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:54:28.412271 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.412178 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls podName:1d264052-c7af-404d-9f73-3a5bed5d755d nodeName:}" failed. No retries permitted until 2026-09-06 05:54:28.912161515 +0000 UTC m=+34.560780674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls") pod "image-registry-854f5cf89c-xtljb" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d") : secret "image-registry-tls" not found Sep 06 05:54:28.412271 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.412219 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecd3697-4253-4467-a57b-368c0950ffe2-config-volume\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.412271 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.412219 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls podName:1ecd3697-4253-4467-a57b-368c0950ffe2 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:28.912203549 +0000 UTC m=+34.560822708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls") pod "dns-default-5g9s6" (UID: "1ecd3697-4253-4467-a57b-368c0950ffe2") : secret "dns-default-metrics-tls" not found Sep 06 05:54:28.412271 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.412261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d264052-c7af-404d-9f73-3a5bed5d755d-installation-pull-secrets\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.412627 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.412289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9b61ebb-d415-4305-abd5-d712c4d0ced3-nginx-conf\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:54:28.412627 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.412318 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-certificates\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.412627 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.412375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d264052-c7af-404d-9f73-3a5bed5d755d-ca-trust-extracted\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.412627 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.412444 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d264052-c7af-404d-9f73-3a5bed5d755d-trusted-ca\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.415697 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.415678 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d264052-c7af-404d-9f73-3a5bed5d755d-installation-pull-secrets\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.415790 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.415684 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d264052-c7af-404d-9f73-3a5bed5d755d-image-registry-private-configuration\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.424601 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.424534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-bound-sa-token\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.424712 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.424632 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbn2m\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-kube-api-access-xbn2m\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.424712 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.424662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwdd\" (UniqueName: \"kubernetes.io/projected/1ecd3697-4253-4467-a57b-368c0950ffe2-kube-api-access-gmwdd\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.512987 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.512948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns46\" (UniqueName: \"kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46\") pod \"network-check-target-jlt55\" (UID: \"32d81600-7fda-434c-8df0-e43a6c3bd84c\") " pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:28.513172 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.513000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4354b120-a2f7-43f8-b3e7-be55e37d543f-cert\") pod \"ingress-canary-x9r8b\" (UID: \"4354b120-a2f7-43f8-b3e7-be55e37d543f\") " pod="openshift-ingress-canary/ingress-canary-x9r8b" Sep 06 05:54:28.513172 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.513066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:54:28.513172 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.513116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2s6w\" (UniqueName: \"kubernetes.io/projected/4354b120-a2f7-43f8-b3e7-be55e37d543f-kube-api-access-s2s6w\") pod \"ingress-canary-x9r8b\" (UID: \"4354b120-a2f7-43f8-b3e7-be55e37d543f\") " pod="openshift-ingress-canary/ingress-canary-x9r8b" Sep 06 05:54:28.513172 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.513173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9b61ebb-d415-4305-abd5-d712c4d0ced3-nginx-conf\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:54:28.513372 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.513124 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:54:28.513372 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.513221 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:54:28.513372 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.513231 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gns46 for pod openshift-network-diagnostics/network-check-target-jlt55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:28.513372 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.513158 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 05:54:28.513372 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.513290 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46 podName:32d81600-7fda-434c-8df0-e43a6c3bd84c nodeName:}" failed. No retries permitted until 2026-09-06 05:55:00.513274068 +0000 UTC m=+66.161893242 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gns46" (UniqueName: "kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46") pod "network-check-target-jlt55" (UID: "32d81600-7fda-434c-8df0-e43a6c3bd84c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:28.513372 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.513353 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert podName:d9b61ebb-d415-4305-abd5-d712c4d0ced3 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:29.013333923 +0000 UTC m=+34.661953096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7dzz" (UID: "d9b61ebb-d415-4305-abd5-d712c4d0ced3") : secret "networking-console-plugin-cert" not found Sep 06 05:54:28.513755 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.513739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9b61ebb-d415-4305-abd5-d712c4d0ced3-nginx-conf\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:54:28.515560 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.515539 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4354b120-a2f7-43f8-b3e7-be55e37d543f-cert\") pod \"ingress-canary-x9r8b\" (UID: \"4354b120-a2f7-43f8-b3e7-be55e37d543f\") " pod="openshift-ingress-canary/ingress-canary-x9r8b" Sep 06 05:54:28.520636 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.520614 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2s6w\" (UniqueName: \"kubernetes.io/projected/4354b120-a2f7-43f8-b3e7-be55e37d543f-kube-api-access-s2s6w\") pod \"ingress-canary-x9r8b\" (UID: \"4354b120-a2f7-43f8-b3e7-be55e37d543f\") " pod="openshift-ingress-canary/ingress-canary-x9r8b" Sep 06 05:54:28.603405 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.603369 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9r8b" Sep 06 05:54:28.613666 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.613631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:28.613666 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.613673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:28.613894 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.613773 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:28.613894 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.613774 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:28.613894 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.613868 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs podName:e5ada9e9-c4df-486d-9325-b5dcecdd451b nodeName:}" failed. No retries permitted until 2026-09-06 05:55:00.613820579 +0000 UTC m=+66.262439750 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs") pod "network-metrics-daemon-2z6qh" (UID: "e5ada9e9-c4df-486d-9325-b5dcecdd451b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:28.614044 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.613902 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret podName:8e95c4ec-ee79-4651-9f84-6f1bd6660624 nodeName:}" failed. No retries permitted until 2026-09-06 05:55:00.613891522 +0000 UTC m=+66.262510683 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret") pod "global-pull-secret-syncer-nksgl" (UID: "8e95c4ec-ee79-4651-9f84-6f1bd6660624") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:28.915983 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.915953 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:28.916622 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.915998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:28.916622 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.916119 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:54:28.916622 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.916127 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:54:28.916622 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.916135 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-854f5cf89c-xtljb: secret "image-registry-tls" not found Sep 06 05:54:28.916622 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.916187 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls podName:1ecd3697-4253-4467-a57b-368c0950ffe2 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:29.916173021 +0000 UTC m=+35.564792197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls") pod "dns-default-5g9s6" (UID: "1ecd3697-4253-4467-a57b-368c0950ffe2") : secret "dns-default-metrics-tls" not found Sep 06 05:54:28.916622 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:28.916201 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls podName:1d264052-c7af-404d-9f73-3a5bed5d755d nodeName:}" failed. No retries permitted until 2026-09-06 05:54:29.916196167 +0000 UTC m=+35.564815324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls") pod "image-registry-854f5cf89c-xtljb" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d") : secret "image-registry-tls" not found Sep 06 05:54:28.963814 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:28.963740 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9r8b"] Sep 06 05:54:28.970722 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:54:28.970680 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4354b120_a2f7_43f8_b3e7_be55e37d543f.slice/crio-0d0a4546d654fa30e70147b8c1f8a34a0593351576a96f753214ee51ea1000ef WatchSource:0}: Error finding container 0d0a4546d654fa30e70147b8c1f8a34a0593351576a96f753214ee51ea1000ef: Status 404 returned error can't find the container with id 0d0a4546d654fa30e70147b8c1f8a34a0593351576a96f753214ee51ea1000ef Sep 06 05:54:29.016426 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.016399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:54:29.016539 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:29.016519 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 05:54:29.016594 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:29.016570 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert podName:d9b61ebb-d415-4305-abd5-d712c4d0ced3 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:30.016558106 +0000 UTC m=+35.665177275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7dzz" (UID: "d9b61ebb-d415-4305-abd5-d712c4d0ced3") : secret "networking-console-plugin-cert" not found Sep 06 05:54:29.018489 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.018460 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9r8b" event={"ID":"4354b120-a2f7-43f8-b3e7-be55e37d543f","Type":"ContainerStarted","Data":"0d0a4546d654fa30e70147b8c1f8a34a0593351576a96f753214ee51ea1000ef"} Sep 06 05:54:29.021178 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.021156 2580 generic.go:356] "Generic (PLEG): container finished" podID="678e8aa3-c088-4f39-bf3d-f158f45c9aef" containerID="ced16aedb8042b5595ff090b3976b09b718de54a9818d3b276fec25a7e9cd0b5" exitCode=0 Sep 06 05:54:29.021276 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.021188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqfs6" event={"ID":"678e8aa3-c088-4f39-bf3d-f158f45c9aef","Type":"ContainerDied","Data":"ced16aedb8042b5595ff090b3976b09b718de54a9818d3b276fec25a7e9cd0b5"} Sep 06 05:54:29.869588 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.869551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:54:29.869588 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.869585 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:54:29.869800 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.869592 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:54:29.873117 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.873091 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 06 05:54:29.873117 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.873104 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 06 05:54:29.874377 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.874356 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n9n5z\"" Sep 06 05:54:29.874492 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.874387 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 06 05:54:29.874492 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.874464 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 06 05:54:29.874600 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.874406 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9rnw9\"" Sep 06 05:54:29.923557 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.923526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:29.924045 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:29.923581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:29.924045 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:29.923759 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:54:29.924045 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:29.923779 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-854f5cf89c-xtljb: secret "image-registry-tls" not found Sep 06 05:54:29.924045 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:29.923778 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:54:29.924045 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:29.923875 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls podName:1d264052-c7af-404d-9f73-3a5bed5d755d nodeName:}" failed. No retries permitted until 2026-09-06 05:54:31.923850291 +0000 UTC m=+37.572469470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls") pod "image-registry-854f5cf89c-xtljb" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d") : secret "image-registry-tls" not found Sep 06 05:54:29.924045 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:29.923898 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls podName:1ecd3697-4253-4467-a57b-368c0950ffe2 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:31.923888094 +0000 UTC m=+37.572507266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls") pod "dns-default-5g9s6" (UID: "1ecd3697-4253-4467-a57b-368c0950ffe2") : secret "dns-default-metrics-tls" not found Sep 06 05:54:30.024176 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:30.024010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:54:30.024176 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:30.024127 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 05:54:30.024414 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:30.024193 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert podName:d9b61ebb-d415-4305-abd5-d712c4d0ced3 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:32.024173268 +0000 UTC m=+37.672792439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7dzz" (UID: "d9b61ebb-d415-4305-abd5-d712c4d0ced3") : secret "networking-console-plugin-cert" not found Sep 06 05:54:30.027896 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:30.027872 2580 generic.go:356] "Generic (PLEG): container finished" podID="678e8aa3-c088-4f39-bf3d-f158f45c9aef" containerID="02e13b031995dd716dc7096b31e596e690528476c6e3ff70836b51f703a2aec8" exitCode=0 Sep 06 05:54:30.028044 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:30.027911 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqfs6" event={"ID":"678e8aa3-c088-4f39-bf3d-f158f45c9aef","Type":"ContainerDied","Data":"02e13b031995dd716dc7096b31e596e690528476c6e3ff70836b51f703a2aec8"} Sep 06 05:54:31.036298 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:31.036264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqfs6" event={"ID":"678e8aa3-c088-4f39-bf3d-f158f45c9aef","Type":"ContainerStarted","Data":"090497ed92cd73f9f74a71835a70f7f6546ed0af97561bc69db60808cbe328df"} Sep 06 05:54:31.059942 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:31.059892 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cqfs6" podStartSLOduration=7.272060138 podStartE2EDuration="37.059875663s" totalImagesPullingTime="29.787815525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:31.058509298 +0000 UTC m=+36.707128479" watchObservedRunningTime="2026-09-06 05:54:31.059875663 +0000 UTC m=+36.708494842" Sep 06 05:54:31.940603 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:31.940557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:31.940766 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:31.940631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:31.940766 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:31.940710 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:54:31.940766 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:31.940731 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-854f5cf89c-xtljb: secret "image-registry-tls" not found Sep 06 05:54:31.940882 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:31.940770 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:54:31.940882 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:31.940788 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls podName:1d264052-c7af-404d-9f73-3a5bed5d755d nodeName:}" failed. No retries permitted until 2026-09-06 05:54:35.940772669 +0000 UTC m=+41.589391826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls") pod "image-registry-854f5cf89c-xtljb" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d") : secret "image-registry-tls" not found Sep 06 05:54:31.940882 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:31.940848 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls podName:1ecd3697-4253-4467-a57b-368c0950ffe2 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:35.940811404 +0000 UTC m=+41.589430582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls") pod "dns-default-5g9s6" (UID: "1ecd3697-4253-4467-a57b-368c0950ffe2") : secret "dns-default-metrics-tls" not found Sep 06 05:54:32.041604 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:32.041571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:54:32.041981 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:32.041700 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 05:54:32.041981 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:32.041762 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert podName:d9b61ebb-d415-4305-abd5-d712c4d0ced3 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:36.041748217 +0000 UTC m=+41.690367379 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7dzz" (UID: "d9b61ebb-d415-4305-abd5-d712c4d0ced3") : secret "networking-console-plugin-cert" not found Sep 06 05:54:33.042476 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:33.042434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9r8b" event={"ID":"4354b120-a2f7-43f8-b3e7-be55e37d543f","Type":"ContainerStarted","Data":"1fd9e6e92b37d25e95343f4db2a2711ef85119d004b3d02c2a32aa59c8e01d7c"} Sep 06 05:54:33.058325 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:33.058273 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x9r8b" podStartSLOduration=1.7993607219999999 podStartE2EDuration="5.058258102s" totalImagesPullingTime="3.25889738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:33.057870658 +0000 UTC m=+38.706489836" watchObservedRunningTime="2026-09-06 05:54:33.058258102 +0000 UTC m=+38.706877280" Sep 06 05:54:35.971326 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:35.971280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:35.971739 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:35.971337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:35.971739 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:35.971444 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:54:35.971739 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:35.971467 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-854f5cf89c-xtljb: secret "image-registry-tls" not found Sep 06 05:54:35.971739 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:35.971483 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:54:35.971739 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:35.971540 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls podName:1d264052-c7af-404d-9f73-3a5bed5d755d nodeName:}" failed. No retries permitted until 2026-09-06 05:54:43.971521656 +0000 UTC m=+49.620140813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls") pod "image-registry-854f5cf89c-xtljb" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d") : secret "image-registry-tls" not found Sep 06 05:54:35.971739 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:35.971556 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls podName:1ecd3697-4253-4467-a57b-368c0950ffe2 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:43.97154859 +0000 UTC m=+49.620167747 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls") pod "dns-default-5g9s6" (UID: "1ecd3697-4253-4467-a57b-368c0950ffe2") : secret "dns-default-metrics-tls" not found Sep 06 05:54:36.071904 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:36.071870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:54:36.072067 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:36.072009 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 05:54:36.072106 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:36.072070 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert podName:d9b61ebb-d415-4305-abd5-d712c4d0ced3 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:44.072052653 +0000 UTC m=+49.720671824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7dzz" (UID: "d9b61ebb-d415-4305-abd5-d712c4d0ced3") : secret "networking-console-plugin-cert" not found Sep 06 05:54:44.039154 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:44.039110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:54:44.039638 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:44.039170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:54:44.039638 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:44.039302 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:54:44.039638 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:44.039362 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls podName:1ecd3697-4253-4467-a57b-368c0950ffe2 nodeName:}" failed. No retries permitted until 2026-09-06 05:55:00.039346494 +0000 UTC m=+65.687965651 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls") pod "dns-default-5g9s6" (UID: "1ecd3697-4253-4467-a57b-368c0950ffe2") : secret "dns-default-metrics-tls" not found Sep 06 05:54:44.039638 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:44.039303 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:54:44.039638 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:44.039402 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-854f5cf89c-xtljb: secret "image-registry-tls" not found Sep 06 05:54:44.039638 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:44.039446 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls podName:1d264052-c7af-404d-9f73-3a5bed5d755d nodeName:}" failed. No retries permitted until 2026-09-06 05:55:00.039432968 +0000 UTC m=+65.688052143 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls") pod "image-registry-854f5cf89c-xtljb" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d") : secret "image-registry-tls" not found Sep 06 05:54:44.140204 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:44.140160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:54:44.140364 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:44.140307 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 05:54:44.140403 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:54:44.140376 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert podName:d9b61ebb-d415-4305-abd5-d712c4d0ced3 nodeName:}" failed. No retries permitted until 2026-09-06 05:55:00.140358073 +0000 UTC m=+65.788977230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7dzz" (UID: "d9b61ebb-d415-4305-abd5-d712c4d0ced3") : secret "networking-console-plugin-cert" not found Sep 06 05:54:55.018200 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:54:55.018172 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-84rs9" Sep 06 05:55:00.063707 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.063656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:55:00.064130 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.063717 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:55:00.064130 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:00.063791 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:55:00.064130 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:00.063812 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-854f5cf89c-xtljb: secret "image-registry-tls" not found Sep 06 05:55:00.064130 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:00.063816 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:55:00.064130 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:00.063892 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls podName:1d264052-c7af-404d-9f73-3a5bed5d755d nodeName:}" failed. No retries permitted until 2026-09-06 05:55:32.063874561 +0000 UTC m=+97.712493718 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls") pod "image-registry-854f5cf89c-xtljb" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d") : secret "image-registry-tls" not found Sep 06 05:55:00.064130 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:00.063906 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls podName:1ecd3697-4253-4467-a57b-368c0950ffe2 nodeName:}" failed. No retries permitted until 2026-09-06 05:55:32.063900477 +0000 UTC m=+97.712519633 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls") pod "dns-default-5g9s6" (UID: "1ecd3697-4253-4467-a57b-368c0950ffe2") : secret "dns-default-metrics-tls" not found Sep 06 05:55:00.164573 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.164533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:55:00.164742 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:00.164690 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 05:55:00.164782 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:00.164754 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert podName:d9b61ebb-d415-4305-abd5-d712c4d0ced3 nodeName:}" failed. No retries permitted until 2026-09-06 05:55:32.164738704 +0000 UTC m=+97.813357861 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7dzz" (UID: "d9b61ebb-d415-4305-abd5-d712c4d0ced3") : secret "networking-console-plugin-cert" not found Sep 06 05:55:00.567909 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.567863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns46\" (UniqueName: \"kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46\") pod \"network-check-target-jlt55\" (UID: \"32d81600-7fda-434c-8df0-e43a6c3bd84c\") " pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:55:00.570879 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.570853 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 06 05:55:00.580938 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.580909 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 06 05:55:00.592416 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.592388 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gns46\" (UniqueName: \"kubernetes.io/projected/32d81600-7fda-434c-8df0-e43a6c3bd84c-kube-api-access-gns46\") pod \"network-check-target-jlt55\" (UID: \"32d81600-7fda-434c-8df0-e43a6c3bd84c\") " pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:55:00.668790 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.668748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:55:00.668790 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.668796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:55:00.671553 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.671525 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 06 05:55:00.671693 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.671676 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 06 05:55:00.679710 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:00.679684 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 06 05:55:00.679808 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:00.679760 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs podName:e5ada9e9-c4df-486d-9325-b5dcecdd451b nodeName:}" failed. No retries permitted until 2026-09-06 05:56:04.679744476 +0000 UTC m=+130.328363646 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs") pod "network-metrics-daemon-2z6qh" (UID: "e5ada9e9-c4df-486d-9325-b5dcecdd451b") : secret "metrics-daemon-secret" not found Sep 06 05:55:00.681801 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.681781 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e95c4ec-ee79-4651-9f84-6f1bd6660624-original-pull-secret\") pod \"global-pull-secret-syncer-nksgl\" (UID: \"8e95c4ec-ee79-4651-9f84-6f1bd6660624\") " pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:55:00.791064 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.791028 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n9n5z\"" Sep 06 05:55:00.794103 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.794087 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nksgl" Sep 06 05:55:00.799706 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:00.799688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:55:01.063710 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:01.063674 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nksgl"] Sep 06 05:55:01.066178 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:01.066158 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jlt55"] Sep 06 05:55:01.068090 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:55:01.068058 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e95c4ec_ee79_4651_9f84_6f1bd6660624.slice/crio-4751f928b47583a24cabb166947528ac62d2b2437ae36e1e5b12f97e2a2df886 WatchSource:0}: Error finding container 4751f928b47583a24cabb166947528ac62d2b2437ae36e1e5b12f97e2a2df886: Status 404 returned error can't find the container with id 4751f928b47583a24cabb166947528ac62d2b2437ae36e1e5b12f97e2a2df886 Sep 06 05:55:01.069027 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:55:01.069007 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d81600_7fda_434c_8df0_e43a6c3bd84c.slice/crio-016d9b312e01c0cbd455535e33a3dd81360a8fd2bb98d9ff21d5f74e5d99dff0 WatchSource:0}: Error finding container 016d9b312e01c0cbd455535e33a3dd81360a8fd2bb98d9ff21d5f74e5d99dff0: Status 404 returned error can't find the container with id 016d9b312e01c0cbd455535e33a3dd81360a8fd2bb98d9ff21d5f74e5d99dff0 Sep 06 05:55:01.109950 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:01.109866 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jlt55" event={"ID":"32d81600-7fda-434c-8df0-e43a6c3bd84c","Type":"ContainerStarted","Data":"016d9b312e01c0cbd455535e33a3dd81360a8fd2bb98d9ff21d5f74e5d99dff0"} Sep 06 05:55:01.110999 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:01.110973 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nksgl" event={"ID":"8e95c4ec-ee79-4651-9f84-6f1bd6660624","Type":"ContainerStarted","Data":"4751f928b47583a24cabb166947528ac62d2b2437ae36e1e5b12f97e2a2df886"} Sep 06 05:55:05.122407 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:05.122370 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jlt55" event={"ID":"32d81600-7fda-434c-8df0-e43a6c3bd84c","Type":"ContainerStarted","Data":"364a7cd613dd968adeda9588dea5f562bf983e59a0b689fdd8e638760a88a6d0"} Sep 06 05:55:05.122790 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:05.122503 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:55:05.138932 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:05.138883 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jlt55" podStartSLOduration=68.029351012 podStartE2EDuration="1m11.138866071s" totalImagesPullingTime="3.109515059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:55:05.137574868 +0000 UTC m=+70.786194045" watchObservedRunningTime="2026-09-06 05:55:05.138866071 +0000 UTC m=+70.787485249" Sep 06 05:55:09.134988 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:09.134942 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nksgl" event={"ID":"8e95c4ec-ee79-4651-9f84-6f1bd6660624","Type":"ContainerStarted","Data":"4f9496d36c43a48b89d63fa0bf16c7dd968bc0aa542b52d5fd5f7dfb7742b5a7"} Sep 06 05:55:09.151538 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:09.151486 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nksgl" podStartSLOduration=66.251024889 podStartE2EDuration="1m14.151472003s" totalImagesPullingTime="7.900447114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:55:09.150824385 +0000 UTC m=+74.799443565" watchObservedRunningTime="2026-09-06 05:55:09.151472003 +0000 UTC m=+74.800091182" Sep 06 05:55:32.111617 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:32.111560 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:55:32.111617 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:32.111631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:55:32.112037 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:32.111722 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:55:32.112037 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:32.111744 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-854f5cf89c-xtljb: secret "image-registry-tls" not found Sep 06 05:55:32.112037 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:32.111751 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:55:32.112037 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:32.111806 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls podName:1d264052-c7af-404d-9f73-3a5bed5d755d nodeName:}" failed. No retries permitted until 2026-09-06 05:56:36.111790303 +0000 UTC m=+161.760409475 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls") pod "image-registry-854f5cf89c-xtljb" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d") : secret "image-registry-tls" not found Sep 06 05:55:32.112037 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:32.111819 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls podName:1ecd3697-4253-4467-a57b-368c0950ffe2 nodeName:}" failed. No retries permitted until 2026-09-06 05:56:36.111813599 +0000 UTC m=+161.760432756 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls") pod "dns-default-5g9s6" (UID: "1ecd3697-4253-4467-a57b-368c0950ffe2") : secret "dns-default-metrics-tls" not found Sep 06 05:55:32.212392 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:32.212352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:55:32.212595 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:32.212518 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 05:55:32.212660 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:55:32.212597 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert podName:d9b61ebb-d415-4305-abd5-d712c4d0ced3 nodeName:}" failed. No retries permitted until 2026-09-06 05:56:36.212576463 +0000 UTC m=+161.861195632 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7dzz" (UID: "d9b61ebb-d415-4305-abd5-d712c4d0ced3") : secret "networking-console-plugin-cert" not found Sep 06 05:55:36.127528 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:55:36.127492 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jlt55" Sep 06 05:56:04.735889 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:56:04.735841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:56:04.736373 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:04.735999 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 06 05:56:04.736373 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:04.736075 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs podName:e5ada9e9-c4df-486d-9325-b5dcecdd451b nodeName:}" failed. No retries permitted until 2026-09-06 05:58:06.73605879 +0000 UTC m=+252.384677951 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs") pod "network-metrics-daemon-2z6qh" (UID: "e5ada9e9-c4df-486d-9325-b5dcecdd451b") : secret "metrics-daemon-secret" not found Sep 06 05:56:31.242463 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:31.242415 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" podUID="1d264052-c7af-404d-9f73-3a5bed5d755d" Sep 06 05:56:31.259555 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:31.259522 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-5g9s6" podUID="1ecd3697-4253-4467-a57b-368c0950ffe2" Sep 06 05:56:31.279702 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:31.279659 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" podUID="d9b61ebb-d415-4305-abd5-d712c4d0ced3" Sep 06 05:56:31.334839 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:56:31.334807 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5g9s6" Sep 06 05:56:31.334966 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:56:31.334853 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:56:32.881656 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:32.881614 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:56:36.162748 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:56:36.162706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:56:36.163168 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:56:36.162757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:56:36.163168 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:36.162883 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:56:36.163168 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:36.162887 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:56:36.163168 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:36.162906 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-854f5cf89c-xtljb: secret "image-registry-tls" not found Sep 06 05:56:36.163168 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:36.162936 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls podName:1ecd3697-4253-4467-a57b-368c0950ffe2 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:38.16292169 +0000 UTC m=+283.811540848 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls") pod "dns-default-5g9s6" (UID: "1ecd3697-4253-4467-a57b-368c0950ffe2") : secret "dns-default-metrics-tls" not found Sep 06 05:56:36.163168 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:36.162949 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls podName:1d264052-c7af-404d-9f73-3a5bed5d755d nodeName:}" failed. No retries permitted until 2026-09-06 05:58:38.16294366 +0000 UTC m=+283.811562817 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls") pod "image-registry-854f5cf89c-xtljb" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d") : secret "image-registry-tls" not found Sep 06 05:56:36.263693 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:56:36.263652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:56:36.263883 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:36.263813 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 05:56:36.263935 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:56:36.263888 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert podName:d9b61ebb-d415-4305-abd5-d712c4d0ced3 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:38.263872228 +0000 UTC m=+283.912491390 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7dzz" (UID: "d9b61ebb-d415-4305-abd5-d712c4d0ced3") : secret "networking-console-plugin-cert" not found Sep 06 05:56:43.869284 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:56:43.869222 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:56:45.868747 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:56:45.868710 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:58:06.800344 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:06.800298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:58:06.800861 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:06.800447 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 06 05:58:06.800861 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:06.800508 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs podName:e5ada9e9-c4df-486d-9325-b5dcecdd451b nodeName:}" failed. No retries permitted until 2026-09-06 06:00:08.800490535 +0000 UTC m=+374.449109699 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs") pod "network-metrics-daemon-2z6qh" (UID: "e5ada9e9-c4df-486d-9325-b5dcecdd451b") : secret "metrics-daemon-secret" not found Sep 06 05:58:22.661753 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.661668 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r5h5s"] Sep 06 05:58:22.664536 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.664514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r5h5s" Sep 06 05:58:22.664656 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.664581 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6"] Sep 06 05:58:22.667322 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.667296 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 06 05:58:22.667460 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.667305 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rw49n\"" Sep 06 05:58:22.667460 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.667304 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 06 05:58:22.667460 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.667440 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx"] Sep 06 05:58:22.667614 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.667514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" Sep 06 05:58:22.670048 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.670030 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n"] Sep 06 05:58:22.670166 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.670136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:22.671518 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.671499 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 06 05:58:22.671951 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.671935 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 06 05:58:22.672032 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.671983 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nx5g4\"" Sep 06 05:58:22.672088 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.672078 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 06 05:58:22.672150 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.672092 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 06 05:58:22.672682 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.672664 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mrdrt\"" Sep 06 05:58:22.672797 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.672708 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:22.672797 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.672766 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 06 05:58:22.673147 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.673131 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 06 05:58:22.673202 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.673153 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 06 05:58:22.675215 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.675182 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 06 05:58:22.675364 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.675346 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 06 05:58:22.675468 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.675404 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 06 05:58:22.675606 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.675503 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-k7w49\"" Sep 06 05:58:22.675820 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.675804 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 06 05:58:22.676457 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.676436 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r5h5s"] Sep 06 05:58:22.678883 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.678761 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx"] Sep 06 05:58:22.681761 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.681737 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6"] Sep 06 05:58:22.691605 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.691587 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n"] Sep 06 05:58:22.713063 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.713034 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-r827n\" (UID: \"c08a861d-896f-4f07-9cb1-96fab97d39ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:22.713185 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.713074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pswqm\" (UniqueName: \"kubernetes.io/projected/eb6157db-5f6c-49a2-afd6-732fae965430-kube-api-access-pswqm\") pod \"volume-data-source-validator-695c895bd6-r5h5s\" (UID: \"eb6157db-5f6c-49a2-afd6-732fae965430\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r5h5s" Sep 06 05:58:22.713185 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.713098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1f02946-f574-44dd-be9e-bb9fbaef6be6-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:22.713185 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.713152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:22.713287 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.713224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x94r6\" (UniqueName: \"kubernetes.io/projected/b1f02946-f574-44dd-be9e-bb9fbaef6be6-kube-api-access-x94r6\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:22.713287 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.713245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78a98a4b-8e37-43cb-9441-90ed382a2e1a-config\") pod \"service-ca-operator-859f94c4b-zmbl6\" (UID: \"78a98a4b-8e37-43cb-9441-90ed382a2e1a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" Sep 06 05:58:22.713287 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.713274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpzxc\" (UniqueName: \"kubernetes.io/projected/78a98a4b-8e37-43cb-9441-90ed382a2e1a-kube-api-access-bpzxc\") pod \"service-ca-operator-859f94c4b-zmbl6\" (UID: \"78a98a4b-8e37-43cb-9441-90ed382a2e1a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" Sep 06 05:58:22.713376 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.713291 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m77ng\" (UniqueName: \"kubernetes.io/projected/c08a861d-896f-4f07-9cb1-96fab97d39ee-kube-api-access-m77ng\") pod \"cluster-samples-operator-7cdcfcf76c-r827n\" (UID: \"c08a861d-896f-4f07-9cb1-96fab97d39ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:22.713376 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.713318 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78a98a4b-8e37-43cb-9441-90ed382a2e1a-serving-cert\") pod \"service-ca-operator-859f94c4b-zmbl6\" (UID: \"78a98a4b-8e37-43cb-9441-90ed382a2e1a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" Sep 06 05:58:22.814467 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.814427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pswqm\" (UniqueName: \"kubernetes.io/projected/eb6157db-5f6c-49a2-afd6-732fae965430-kube-api-access-pswqm\") pod \"volume-data-source-validator-695c895bd6-r5h5s\" (UID: \"eb6157db-5f6c-49a2-afd6-732fae965430\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r5h5s" Sep 06 05:58:22.814467 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.814469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1f02946-f574-44dd-be9e-bb9fbaef6be6-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:22.814703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.814495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:22.814703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.814541 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94r6\" (UniqueName: \"kubernetes.io/projected/b1f02946-f574-44dd-be9e-bb9fbaef6be6-kube-api-access-x94r6\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:22.814703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.814566 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78a98a4b-8e37-43cb-9441-90ed382a2e1a-config\") pod \"service-ca-operator-859f94c4b-zmbl6\" (UID: \"78a98a4b-8e37-43cb-9441-90ed382a2e1a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" Sep 06 05:58:22.814703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.814589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzxc\" (UniqueName: \"kubernetes.io/projected/78a98a4b-8e37-43cb-9441-90ed382a2e1a-kube-api-access-bpzxc\") pod \"service-ca-operator-859f94c4b-zmbl6\" (UID: \"78a98a4b-8e37-43cb-9441-90ed382a2e1a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" Sep 06 05:58:22.814703 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.814612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m77ng\" (UniqueName: \"kubernetes.io/projected/c08a861d-896f-4f07-9cb1-96fab97d39ee-kube-api-access-m77ng\") pod \"cluster-samples-operator-7cdcfcf76c-r827n\" (UID: \"c08a861d-896f-4f07-9cb1-96fab97d39ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:22.814703 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:22.814654 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 06 05:58:22.815000 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:22.814728 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls podName:b1f02946-f574-44dd-be9e-bb9fbaef6be6 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:23.314706071 +0000 UTC m=+268.963325247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-bh6dx" (UID: "b1f02946-f574-44dd-be9e-bb9fbaef6be6") : secret "cluster-monitoring-operator-tls" not found Sep 06 05:58:22.815000 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.814769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78a98a4b-8e37-43cb-9441-90ed382a2e1a-serving-cert\") pod \"service-ca-operator-859f94c4b-zmbl6\" (UID: \"78a98a4b-8e37-43cb-9441-90ed382a2e1a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" Sep 06 05:58:22.815000 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.814822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-r827n\" (UID: \"c08a861d-896f-4f07-9cb1-96fab97d39ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:22.815000 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:22.814950 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 06 05:58:22.815000 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:22.814993 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls podName:c08a861d-896f-4f07-9cb1-96fab97d39ee nodeName:}" failed. No retries permitted until 2026-09-06 05:58:23.314979271 +0000 UTC m=+268.963598428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-r827n" (UID: "c08a861d-896f-4f07-9cb1-96fab97d39ee") : secret "samples-operator-tls" not found Sep 06 05:58:22.815225 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.815207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78a98a4b-8e37-43cb-9441-90ed382a2e1a-config\") pod \"service-ca-operator-859f94c4b-zmbl6\" (UID: \"78a98a4b-8e37-43cb-9441-90ed382a2e1a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" Sep 06 05:58:22.815266 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.815234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1f02946-f574-44dd-be9e-bb9fbaef6be6-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:22.817014 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.816993 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78a98a4b-8e37-43cb-9441-90ed382a2e1a-serving-cert\") pod \"service-ca-operator-859f94c4b-zmbl6\" (UID: \"78a98a4b-8e37-43cb-9441-90ed382a2e1a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" Sep 06 05:58:22.827569 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.827548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzxc\" (UniqueName: \"kubernetes.io/projected/78a98a4b-8e37-43cb-9441-90ed382a2e1a-kube-api-access-bpzxc\") pod \"service-ca-operator-859f94c4b-zmbl6\" (UID: \"78a98a4b-8e37-43cb-9441-90ed382a2e1a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" Sep 06 05:58:22.827670 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.827569 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94r6\" (UniqueName: \"kubernetes.io/projected/b1f02946-f574-44dd-be9e-bb9fbaef6be6-kube-api-access-x94r6\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:22.827670 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.827638 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pswqm\" (UniqueName: \"kubernetes.io/projected/eb6157db-5f6c-49a2-afd6-732fae965430-kube-api-access-pswqm\") pod \"volume-data-source-validator-695c895bd6-r5h5s\" (UID: \"eb6157db-5f6c-49a2-afd6-732fae965430\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r5h5s" Sep 06 05:58:22.827794 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.827743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m77ng\" (UniqueName: \"kubernetes.io/projected/c08a861d-896f-4f07-9cb1-96fab97d39ee-kube-api-access-m77ng\") pod \"cluster-samples-operator-7cdcfcf76c-r827n\" (UID: \"c08a861d-896f-4f07-9cb1-96fab97d39ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:22.974730 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.974690 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r5h5s" Sep 06 05:58:22.982897 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:22.982868 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" Sep 06 05:58:23.246168 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:23.246144 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r5h5s"] Sep 06 05:58:23.248388 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:23.248364 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6"] Sep 06 05:58:23.248759 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:23.248736 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6157db_5f6c_49a2_afd6_732fae965430.slice/crio-e0afca1930bdead30fd74ec963d0f8d6d4b8bc2b8a2e0ab34ac6f71de99b06ec WatchSource:0}: Error finding container e0afca1930bdead30fd74ec963d0f8d6d4b8bc2b8a2e0ab34ac6f71de99b06ec: Status 404 returned error can't find the container with id e0afca1930bdead30fd74ec963d0f8d6d4b8bc2b8a2e0ab34ac6f71de99b06ec Sep 06 05:58:23.250989 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:23.250924 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a98a4b_8e37_43cb_9441_90ed382a2e1a.slice/crio-306b896c9ef4ebaf464f04e608b1d85e071b77906992cda3bb9c63c13937190c WatchSource:0}: Error finding container 306b896c9ef4ebaf464f04e608b1d85e071b77906992cda3bb9c63c13937190c: Status 404 returned error can't find the container with id 306b896c9ef4ebaf464f04e608b1d85e071b77906992cda3bb9c63c13937190c Sep 06 05:58:23.318482 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:23.318451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-r827n\" (UID: \"c08a861d-896f-4f07-9cb1-96fab97d39ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:23.318607 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:23.318500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:23.318607 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:23.318594 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 06 05:58:23.318679 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:23.318653 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls podName:c08a861d-896f-4f07-9cb1-96fab97d39ee nodeName:}" failed. No retries permitted until 2026-09-06 05:58:24.318639538 +0000 UTC m=+269.967258700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-r827n" (UID: "c08a861d-896f-4f07-9cb1-96fab97d39ee") : secret "samples-operator-tls" not found Sep 06 05:58:23.318733 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:23.318594 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 06 05:58:23.318733 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:23.318717 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls podName:b1f02946-f574-44dd-be9e-bb9fbaef6be6 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:24.31870525 +0000 UTC m=+269.967324410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-bh6dx" (UID: "b1f02946-f574-44dd-be9e-bb9fbaef6be6") : secret "cluster-monitoring-operator-tls" not found Sep 06 05:58:23.601840 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:23.601734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" event={"ID":"78a98a4b-8e37-43cb-9441-90ed382a2e1a","Type":"ContainerStarted","Data":"306b896c9ef4ebaf464f04e608b1d85e071b77906992cda3bb9c63c13937190c"} Sep 06 05:58:23.602740 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:23.602718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r5h5s" event={"ID":"eb6157db-5f6c-49a2-afd6-732fae965430","Type":"ContainerStarted","Data":"e0afca1930bdead30fd74ec963d0f8d6d4b8bc2b8a2e0ab34ac6f71de99b06ec"} Sep 06 05:58:24.328156 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:24.328120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-r827n\" (UID: \"c08a861d-896f-4f07-9cb1-96fab97d39ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:24.328615 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:24.328196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:24.328615 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:24.328304 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 06 05:58:24.328615 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:24.328318 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 06 05:58:24.328615 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:24.328380 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls podName:b1f02946-f574-44dd-be9e-bb9fbaef6be6 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:26.328359703 +0000 UTC m=+271.976978864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-bh6dx" (UID: "b1f02946-f574-44dd-be9e-bb9fbaef6be6") : secret "cluster-monitoring-operator-tls" not found Sep 06 05:58:24.328615 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:24.328400 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls podName:c08a861d-896f-4f07-9cb1-96fab97d39ee nodeName:}" failed. No retries permitted until 2026-09-06 05:58:26.328390314 +0000 UTC m=+271.977009471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-r827n" (UID: "c08a861d-896f-4f07-9cb1-96fab97d39ee") : secret "samples-operator-tls" not found Sep 06 05:58:24.607213 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:24.607138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r5h5s" event={"ID":"eb6157db-5f6c-49a2-afd6-732fae965430","Type":"ContainerStarted","Data":"b53f0500b814eeef7843ee87f4182d1e706adc6348a4fd3d446e14daccddb46f"} Sep 06 05:58:24.626711 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:24.626657 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r5h5s" podStartSLOduration=1.667343973 podStartE2EDuration="2.626644485s" totalImagesPullingTime="959.300512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:24.625852024 +0000 UTC m=+270.274471204" watchObservedRunningTime="2026-09-06 05:58:24.626644485 +0000 UTC m=+270.275263687" Sep 06 05:58:25.611061 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:25.610972 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" event={"ID":"78a98a4b-8e37-43cb-9441-90ed382a2e1a","Type":"ContainerStarted","Data":"33858713598e1823ef40cff07feae15d48f97cc2383cf7fc84834d43d8b8e153"} Sep 06 05:58:25.627470 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:25.627422 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" podStartSLOduration=1.633966005 podStartE2EDuration="3.627406465s" totalImagesPullingTime="1.99344046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:25.625887377 +0000 UTC m=+271.274506560" watchObservedRunningTime="2026-09-06 05:58:25.627406465 +0000 UTC m=+271.276025693" Sep 06 05:58:26.345795 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:26.345753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:26.345974 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:26.345867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-r827n\" (UID: \"c08a861d-896f-4f07-9cb1-96fab97d39ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:26.345974 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:26.345918 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 06 05:58:26.345974 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:26.345967 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 06 05:58:26.346070 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:26.345984 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls podName:b1f02946-f574-44dd-be9e-bb9fbaef6be6 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:30.34596991 +0000 UTC m=+275.994589067 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-bh6dx" (UID: "b1f02946-f574-44dd-be9e-bb9fbaef6be6") : secret "cluster-monitoring-operator-tls" not found Sep 06 05:58:26.346070 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:26.346006 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls podName:c08a861d-896f-4f07-9cb1-96fab97d39ee nodeName:}" failed. No retries permitted until 2026-09-06 05:58:30.345994109 +0000 UTC m=+275.994613265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-r827n" (UID: "c08a861d-896f-4f07-9cb1-96fab97d39ee") : secret "samples-operator-tls" not found Sep 06 05:58:27.516937 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.516903 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tzmc6"] Sep 06 05:58:27.520164 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.520144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.522784 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.522761 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 06 05:58:27.523339 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.523322 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5nmlz\"" Sep 06 05:58:27.524019 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.524003 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 06 05:58:27.524085 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.524022 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 06 05:58:27.524085 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.524008 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 06 05:58:27.536594 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.536575 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tzmc6"] Sep 06 05:58:27.656350 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.656316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/627d4abe-9c46-41be-9d8b-ae68e88ce293-crio-socket\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.656519 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.656440 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.656519 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.656478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2nf\" (UniqueName: \"kubernetes.io/projected/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-api-access-cg2nf\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.656649 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.656549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.656649 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.656592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/627d4abe-9c46-41be-9d8b-ae68e88ce293-data-volume\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.747057 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.747019 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zwkhp"] Sep 06 05:58:27.750078 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.750063 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zwkhp" Sep 06 05:58:27.752635 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.752617 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 06 05:58:27.752943 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.752923 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 06 05:58:27.753037 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.752981 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-x7bks\"" Sep 06 05:58:27.757478 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.757452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.757576 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.757494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2nf\" (UniqueName: \"kubernetes.io/projected/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-api-access-cg2nf\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.757629 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.757588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.757629 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.757620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/627d4abe-9c46-41be-9d8b-ae68e88ce293-data-volume\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.757714 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.757657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/627d4abe-9c46-41be-9d8b-ae68e88ce293-crio-socket\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.757856 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.757818 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/627d4abe-9c46-41be-9d8b-ae68e88ce293-crio-socket\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.757938 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:27.757924 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 05:58:27.758034 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:27.758024 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm podName:627d4abe-9c46-41be-9d8b-ae68e88ce293 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:28.257998605 +0000 UTC m=+273.906617779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tzmc6" (UID: "627d4abe-9c46-41be-9d8b-ae68e88ce293") : configmap "kube-rbac-proxy" not found Sep 06 05:58:27.758534 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:27.758515 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 05:58:27.758604 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:27.758577 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls podName:627d4abe-9c46-41be-9d8b-ae68e88ce293 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:28.258554708 +0000 UTC m=+273.907173870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tzmc6" (UID: "627d4abe-9c46-41be-9d8b-ae68e88ce293") : secret "insights-runtime-extractor-tls" not found Sep 06 05:58:27.758897 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.758878 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/627d4abe-9c46-41be-9d8b-ae68e88ce293-data-volume\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.762297 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.762275 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zwkhp"] Sep 06 05:58:27.769268 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.769216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2nf\" (UniqueName: \"kubernetes.io/projected/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-api-access-cg2nf\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:27.858579 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.858535 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ls77\" (UniqueName: \"kubernetes.io/projected/721c23a4-d6f1-4f87-836e-741f52dced36-kube-api-access-5ls77\") pod \"migrator-fdbb8fdff-zwkhp\" (UID: \"721c23a4-d6f1-4f87-836e-741f52dced36\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zwkhp" Sep 06 05:58:27.959636 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.959597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ls77\" (UniqueName: \"kubernetes.io/projected/721c23a4-d6f1-4f87-836e-741f52dced36-kube-api-access-5ls77\") pod \"migrator-fdbb8fdff-zwkhp\" (UID: \"721c23a4-d6f1-4f87-836e-741f52dced36\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zwkhp" Sep 06 05:58:27.967638 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:27.967608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ls77\" (UniqueName: \"kubernetes.io/projected/721c23a4-d6f1-4f87-836e-741f52dced36-kube-api-access-5ls77\") pod \"migrator-fdbb8fdff-zwkhp\" (UID: \"721c23a4-d6f1-4f87-836e-741f52dced36\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zwkhp" Sep 06 05:58:28.063181 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.063097 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zwkhp" Sep 06 05:58:28.261600 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.261557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:28.261780 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.261641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:28.261780 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:28.261681 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 05:58:28.261780 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:28.261751 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm podName:627d4abe-9c46-41be-9d8b-ae68e88ce293 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:29.261732726 +0000 UTC m=+274.910351900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tzmc6" (UID: "627d4abe-9c46-41be-9d8b-ae68e88ce293") : configmap "kube-rbac-proxy" not found Sep 06 05:58:28.261780 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:28.261774 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 05:58:28.261971 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:28.261822 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls podName:627d4abe-9c46-41be-9d8b-ae68e88ce293 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:29.261811318 +0000 UTC m=+274.910430482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tzmc6" (UID: "627d4abe-9c46-41be-9d8b-ae68e88ce293") : secret "insights-runtime-extractor-tls" not found Sep 06 05:58:28.328105 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.328081 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zwkhp"] Sep 06 05:58:28.330179 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:28.330155 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721c23a4_d6f1_4f87_836e_741f52dced36.slice/crio-025552fc7b3324221cc72af733bef638d0b121a3dcc970377dc32065d44590e3 WatchSource:0}: Error finding container 025552fc7b3324221cc72af733bef638d0b121a3dcc970377dc32065d44590e3: Status 404 returned error can't find the container with id 025552fc7b3324221cc72af733bef638d0b121a3dcc970377dc32065d44590e3 Sep 06 05:58:28.619940 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.619858 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zwkhp" event={"ID":"721c23a4-d6f1-4f87-836e-741f52dced36","Type":"ContainerStarted","Data":"025552fc7b3324221cc72af733bef638d0b121a3dcc970377dc32065d44590e3"} Sep 06 05:58:28.765683 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.765651 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-d4jkl"] Sep 06 05:58:28.769663 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.769645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-d4jkl" Sep 06 05:58:28.772467 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.772438 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-h67wr\"" Sep 06 05:58:28.772467 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.772438 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 06 05:58:28.772615 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.772537 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 06 05:58:28.773613 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.773592 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 06 05:58:28.773715 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.773596 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 06 05:58:28.777512 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.777492 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-d4jkl"] Sep 06 05:58:28.865485 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.865445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e28bff64-6311-4b3e-96ef-7296c3b43180-signing-cabundle\") pod \"service-ca-75d99c484-d4jkl\" (UID: \"e28bff64-6311-4b3e-96ef-7296c3b43180\") " pod="openshift-service-ca/service-ca-75d99c484-d4jkl" Sep 06 05:58:28.865485 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.865487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vbqj\" (UniqueName: \"kubernetes.io/projected/e28bff64-6311-4b3e-96ef-7296c3b43180-kube-api-access-5vbqj\") pod \"service-ca-75d99c484-d4jkl\" (UID: \"e28bff64-6311-4b3e-96ef-7296c3b43180\") " pod="openshift-service-ca/service-ca-75d99c484-d4jkl" Sep 06 05:58:28.865714 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.865579 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e28bff64-6311-4b3e-96ef-7296c3b43180-signing-key\") pod \"service-ca-75d99c484-d4jkl\" (UID: \"e28bff64-6311-4b3e-96ef-7296c3b43180\") " pod="openshift-service-ca/service-ca-75d99c484-d4jkl" Sep 06 05:58:28.966744 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.966706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e28bff64-6311-4b3e-96ef-7296c3b43180-signing-cabundle\") pod \"service-ca-75d99c484-d4jkl\" (UID: \"e28bff64-6311-4b3e-96ef-7296c3b43180\") " pod="openshift-service-ca/service-ca-75d99c484-d4jkl" Sep 06 05:58:28.966744 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.966752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbqj\" (UniqueName: \"kubernetes.io/projected/e28bff64-6311-4b3e-96ef-7296c3b43180-kube-api-access-5vbqj\") pod \"service-ca-75d99c484-d4jkl\" (UID: \"e28bff64-6311-4b3e-96ef-7296c3b43180\") " pod="openshift-service-ca/service-ca-75d99c484-d4jkl" Sep 06 05:58:28.967009 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.966797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e28bff64-6311-4b3e-96ef-7296c3b43180-signing-key\") pod \"service-ca-75d99c484-d4jkl\" (UID: \"e28bff64-6311-4b3e-96ef-7296c3b43180\") " pod="openshift-service-ca/service-ca-75d99c484-d4jkl" Sep 06 05:58:28.967528 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.967497 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e28bff64-6311-4b3e-96ef-7296c3b43180-signing-cabundle\") pod \"service-ca-75d99c484-d4jkl\" (UID: \"e28bff64-6311-4b3e-96ef-7296c3b43180\") " pod="openshift-service-ca/service-ca-75d99c484-d4jkl" Sep 06 05:58:28.969955 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.969931 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e28bff64-6311-4b3e-96ef-7296c3b43180-signing-key\") pod \"service-ca-75d99c484-d4jkl\" (UID: \"e28bff64-6311-4b3e-96ef-7296c3b43180\") " pod="openshift-service-ca/service-ca-75d99c484-d4jkl" Sep 06 05:58:28.975643 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:28.975618 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbqj\" (UniqueName: \"kubernetes.io/projected/e28bff64-6311-4b3e-96ef-7296c3b43180-kube-api-access-5vbqj\") pod \"service-ca-75d99c484-d4jkl\" (UID: \"e28bff64-6311-4b3e-96ef-7296c3b43180\") " pod="openshift-service-ca/service-ca-75d99c484-d4jkl" Sep 06 05:58:29.079017 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:29.078981 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-d4jkl" Sep 06 05:58:29.268885 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:29.268775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:29.269046 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:29.268893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:29.269046 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:29.268923 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 05:58:29.269046 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:29.268985 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 05:58:29.269046 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:29.269004 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm podName:627d4abe-9c46-41be-9d8b-ae68e88ce293 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:31.268982551 +0000 UTC m=+276.917601732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tzmc6" (UID: "627d4abe-9c46-41be-9d8b-ae68e88ce293") : configmap "kube-rbac-proxy" not found Sep 06 05:58:29.269262 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:29.269057 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls podName:627d4abe-9c46-41be-9d8b-ae68e88ce293 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:31.269026735 +0000 UTC m=+276.917645892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tzmc6" (UID: "627d4abe-9c46-41be-9d8b-ae68e88ce293") : secret "insights-runtime-extractor-tls" not found Sep 06 05:58:29.352428 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:29.352244 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-d4jkl"] Sep 06 05:58:29.374768 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:29.374740 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28bff64_6311_4b3e_96ef_7296c3b43180.slice/crio-ef7a1c23e4640ff3558ade2320853cbc5bbf65db393351aad63a88b8ff242209 WatchSource:0}: Error finding container ef7a1c23e4640ff3558ade2320853cbc5bbf65db393351aad63a88b8ff242209: Status 404 returned error can't find the container with id ef7a1c23e4640ff3558ade2320853cbc5bbf65db393351aad63a88b8ff242209 Sep 06 05:58:29.624798 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:29.624703 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zwkhp" event={"ID":"721c23a4-d6f1-4f87-836e-741f52dced36","Type":"ContainerStarted","Data":"6a0ecbffefc0ddfea092fa73949d89728a6923314eb2b966d17f9b97b7516a45"} Sep 06 05:58:29.624798 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:29.624767 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zwkhp" event={"ID":"721c23a4-d6f1-4f87-836e-741f52dced36","Type":"ContainerStarted","Data":"9bfa3c7e525fb3ed28b7fef99b8e120d1483e375dba82c7f5e73df6697b8569c"} Sep 06 05:58:29.626165 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:29.626140 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-d4jkl" event={"ID":"e28bff64-6311-4b3e-96ef-7296c3b43180","Type":"ContainerStarted","Data":"8faa3f2da22c16e0b60934b9feee6d5e14e48362b62b01d2b6626925026a96d4"} Sep 06 05:58:29.626253 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:29.626178 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-d4jkl" event={"ID":"e28bff64-6311-4b3e-96ef-7296c3b43180","Type":"ContainerStarted","Data":"ef7a1c23e4640ff3558ade2320853cbc5bbf65db393351aad63a88b8ff242209"} Sep 06 05:58:29.641629 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:29.641582 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zwkhp" podStartSLOduration=1.5925599529999999 podStartE2EDuration="2.641568422s" totalImagesPullingTime="1.049008469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:29.640987146 +0000 UTC m=+275.289606325" watchObservedRunningTime="2026-09-06 05:58:29.641568422 +0000 UTC m=+275.290187642" Sep 06 05:58:29.657890 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:29.657847 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-d4jkl" podStartSLOduration=1.6578228130000001 podStartE2EDuration="1.657822813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:29.657332952 +0000 UTC m=+275.305952131" watchObservedRunningTime="2026-09-06 05:58:29.657822813 +0000 UTC m=+275.306441991" Sep 06 05:58:29.666139 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:29.666110 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xxk4z_27b14080-015b-496f-b375-cc4cbdbf1892/dns-node-resolver/0.log" Sep 06 05:58:30.378711 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:30.378675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-r827n\" (UID: \"c08a861d-896f-4f07-9cb1-96fab97d39ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:30.378882 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:30.378753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:30.378882 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:30.378868 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 06 05:58:30.378882 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:30.378872 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 06 05:58:30.378994 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:30.378924 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls podName:b1f02946-f574-44dd-be9e-bb9fbaef6be6 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:38.378908359 +0000 UTC m=+284.027527523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-bh6dx" (UID: "b1f02946-f574-44dd-be9e-bb9fbaef6be6") : secret "cluster-monitoring-operator-tls" not found Sep 06 05:58:30.378994 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:30.378936 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls podName:c08a861d-896f-4f07-9cb1-96fab97d39ee nodeName:}" failed. No retries permitted until 2026-09-06 05:58:38.37893088 +0000 UTC m=+284.027550036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-r827n" (UID: "c08a861d-896f-4f07-9cb1-96fab97d39ee") : secret "samples-operator-tls" not found Sep 06 05:58:30.465244 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:30.465216 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cb5np_ce82693f-a593-41cf-83c0-d09f07d50168/node-ca/0.log" Sep 06 05:58:31.287997 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:31.287955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:31.288450 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:31.288088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:31.288450 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:31.288243 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 05:58:31.288450 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:31.288331 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls podName:627d4abe-9c46-41be-9d8b-ae68e88ce293 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:35.288312619 +0000 UTC m=+280.936931781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tzmc6" (UID: "627d4abe-9c46-41be-9d8b-ae68e88ce293") : secret "insights-runtime-extractor-tls" not found Sep 06 05:58:31.288759 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:31.288739 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 05:58:31.288941 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:31.288919 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm podName:627d4abe-9c46-41be-9d8b-ae68e88ce293 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:35.288897667 +0000 UTC m=+280.937516831 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tzmc6" (UID: "627d4abe-9c46-41be-9d8b-ae68e88ce293") : configmap "kube-rbac-proxy" not found Sep 06 05:58:31.465207 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:31.465170 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9r8b_4354b120-a2f7-43f8-b3e7-be55e37d543f/serve-healthcheck-canary/0.log" Sep 06 05:58:34.335337 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:34.335280 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-5g9s6" podUID="1ecd3697-4253-4467-a57b-368c0950ffe2" Sep 06 05:58:34.335724 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:34.335418 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" podUID="d9b61ebb-d415-4305-abd5-d712c4d0ced3" Sep 06 05:58:34.643719 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:34.643643 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:58:34.643874 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:34.643648 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5g9s6" Sep 06 05:58:35.322284 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:35.322248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:35.322465 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:35.322329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:35.322465 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:35.322376 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 05:58:35.322465 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:35.322446 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm podName:627d4abe-9c46-41be-9d8b-ae68e88ce293 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:43.322426948 +0000 UTC m=+288.971046120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tzmc6" (UID: "627d4abe-9c46-41be-9d8b-ae68e88ce293") : configmap "kube-rbac-proxy" not found Sep 06 05:58:35.322566 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:35.322471 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 05:58:35.322566 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:35.322526 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls podName:627d4abe-9c46-41be-9d8b-ae68e88ce293 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:43.322512483 +0000 UTC m=+288.971131640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tzmc6" (UID: "627d4abe-9c46-41be-9d8b-ae68e88ce293") : secret "insights-runtime-extractor-tls" not found Sep 06 05:58:38.246953 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.246915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:58:38.246953 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.246962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:58:38.249253 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.249230 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecd3697-4253-4467-a57b-368c0950ffe2-metrics-tls\") pod \"dns-default-5g9s6\" (UID: \"1ecd3697-4253-4467-a57b-368c0950ffe2\") " pod="openshift-dns/dns-default-5g9s6" Sep 06 05:58:38.249383 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.249364 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"image-registry-854f5cf89c-xtljb\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:58:38.347704 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.347667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 05:58:38.347915 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:38.347808 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 05:58:38.347915 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:38.347895 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert podName:d9b61ebb-d415-4305-abd5-d712c4d0ced3 nodeName:}" failed. No retries permitted until 2026-09-06 06:00:40.347878942 +0000 UTC m=+405.996498099 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7dzz" (UID: "d9b61ebb-d415-4305-abd5-d712c4d0ced3") : secret "networking-console-plugin-cert" not found Sep 06 05:58:38.448362 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.448313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-r827n\" (UID: \"c08a861d-896f-4f07-9cb1-96fab97d39ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:38.448502 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.448413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:38.450937 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.450911 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c08a861d-896f-4f07-9cb1-96fab97d39ee-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-r827n\" (UID: \"c08a861d-896f-4f07-9cb1-96fab97d39ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:38.451043 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.450912 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1f02946-f574-44dd-be9e-bb9fbaef6be6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-bh6dx\" (UID: \"b1f02946-f574-44dd-be9e-bb9fbaef6be6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:38.472902 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.472875 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qcp4t\"" Sep 06 05:58:38.481155 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.481132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:58:38.547107 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.547079 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ppscw\"" Sep 06 05:58:38.554452 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.554421 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5g9s6" Sep 06 05:58:38.589407 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.589377 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" Sep 06 05:58:38.594722 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.594700 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" Sep 06 05:58:38.752923 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.752894 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-854f5cf89c-xtljb"] Sep 06 05:58:38.755514 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:38.755489 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d264052_c7af_404d_9f73_3a5bed5d755d.slice/crio-439836ce6bafa928774d811b1b50abb2163feec65849f6e01f10855b0639665d WatchSource:0}: Error finding container 439836ce6bafa928774d811b1b50abb2163feec65849f6e01f10855b0639665d: Status 404 returned error can't find the container with id 439836ce6bafa928774d811b1b50abb2163feec65849f6e01f10855b0639665d Sep 06 05:58:38.824487 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.824462 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5g9s6"] Sep 06 05:58:38.827251 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:38.827226 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecd3697_4253_4467_a57b_368c0950ffe2.slice/crio-24de7ffd17895eb57fc3cf48f7cb88d526ff95bc4d0fdc00d6c64f7e092db450 WatchSource:0}: Error finding container 24de7ffd17895eb57fc3cf48f7cb88d526ff95bc4d0fdc00d6c64f7e092db450: Status 404 returned error can't find the container with id 24de7ffd17895eb57fc3cf48f7cb88d526ff95bc4d0fdc00d6c64f7e092db450 Sep 06 05:58:38.865161 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.865131 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx"] Sep 06 05:58:38.868049 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:38.868023 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n"] Sep 06 05:58:38.868638 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:38.868610 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f02946_f574_44dd_be9e_bb9fbaef6be6.slice/crio-52c299344ee274cc13dd4e95e59b2266c41861f452360734eae4b2b5baeb9401 WatchSource:0}: Error finding container 52c299344ee274cc13dd4e95e59b2266c41861f452360734eae4b2b5baeb9401: Status 404 returned error can't find the container with id 52c299344ee274cc13dd4e95e59b2266c41861f452360734eae4b2b5baeb9401 Sep 06 05:58:39.663939 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:39.663894 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" event={"ID":"c08a861d-896f-4f07-9cb1-96fab97d39ee","Type":"ContainerStarted","Data":"76b746ff70b903902503cebc83c05c172aee0875f001d2264586eab6ec77b979"} Sep 06 05:58:39.665809 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:39.665780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5g9s6" event={"ID":"1ecd3697-4253-4467-a57b-368c0950ffe2","Type":"ContainerStarted","Data":"24de7ffd17895eb57fc3cf48f7cb88d526ff95bc4d0fdc00d6c64f7e092db450"} Sep 06 05:58:39.668719 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:39.668691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" event={"ID":"1d264052-c7af-404d-9f73-3a5bed5d755d","Type":"ContainerStarted","Data":"6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f"} Sep 06 05:58:39.668805 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:39.668736 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" event={"ID":"1d264052-c7af-404d-9f73-3a5bed5d755d","Type":"ContainerStarted","Data":"439836ce6bafa928774d811b1b50abb2163feec65849f6e01f10855b0639665d"} Sep 06 05:58:39.669511 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:39.669481 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:58:39.671528 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:39.671462 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" event={"ID":"b1f02946-f574-44dd-be9e-bb9fbaef6be6","Type":"ContainerStarted","Data":"52c299344ee274cc13dd4e95e59b2266c41861f452360734eae4b2b5baeb9401"} Sep 06 05:58:39.690314 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:39.690278 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" podStartSLOduration=284.690266426 podStartE2EDuration="4m44.690266426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:39.689030925 +0000 UTC m=+285.337650114" watchObservedRunningTime="2026-09-06 05:58:39.690266426 +0000 UTC m=+285.338885605" Sep 06 05:58:42.687411 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:42.687374 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" event={"ID":"b1f02946-f574-44dd-be9e-bb9fbaef6be6","Type":"ContainerStarted","Data":"24597c14a4390d70783bfd83fc5b54ee4a964a4d000a487a3f9ad8d25c39eb52"} Sep 06 05:58:42.689203 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:42.689179 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" event={"ID":"c08a861d-896f-4f07-9cb1-96fab97d39ee","Type":"ContainerStarted","Data":"3a9702bf12a7e01a6a7a2c6ed4afc42539fdec80b43794031b1f2f2362b90dfa"} Sep 06 05:58:42.689328 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:42.689225 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" event={"ID":"c08a861d-896f-4f07-9cb1-96fab97d39ee","Type":"ContainerStarted","Data":"8df3ab3bc1fc6f93005b60392a682c41b68e165c89ccebd9fb0f76a64d22e464"} Sep 06 05:58:42.690926 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:42.690904 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5g9s6" event={"ID":"1ecd3697-4253-4467-a57b-368c0950ffe2","Type":"ContainerStarted","Data":"ca71b8f29c9ce1afe99768da6ab8f8e5b6033ad8a6b672d1cbf28f3982d40593"} Sep 06 05:58:42.691058 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:42.690940 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5g9s6" event={"ID":"1ecd3697-4253-4467-a57b-368c0950ffe2","Type":"ContainerStarted","Data":"257668cd8ccdc0714333d8d3f5474c2800ccf7778e2d2275f7601877bb83d602"} Sep 06 05:58:42.691058 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:42.690975 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5g9s6" Sep 06 05:58:42.702542 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:42.702493 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-bh6dx" podStartSLOduration=17.953913272 podStartE2EDuration="20.702481254s" totalImagesPullingTime="2.748567982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:42.701700786 +0000 UTC m=+288.350319965" watchObservedRunningTime="2026-09-06 05:58:42.702481254 +0000 UTC m=+288.351100432" Sep 06 05:58:42.717723 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:42.717673 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-r827n" podStartSLOduration=18.030373804 podStartE2EDuration="20.717659118s" totalImagesPullingTime="2.687285314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:42.716688062 +0000 UTC m=+288.365307241" watchObservedRunningTime="2026-09-06 05:58:42.717659118 +0000 UTC m=+288.366278296" Sep 06 05:58:42.731573 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:42.731531 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5g9s6" podStartSLOduration=251.94308987 podStartE2EDuration="4m14.731516287s" totalImagesPullingTime="2.788426417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:42.730435966 +0000 UTC m=+288.379055157" watchObservedRunningTime="2026-09-06 05:58:42.731516287 +0000 UTC m=+288.380135517" Sep 06 05:58:43.392283 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:43.392232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:43.392283 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:43.392291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:43.392821 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:43.392795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/627d4abe-9c46-41be-9d8b-ae68e88ce293-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:43.394522 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:43.394500 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/627d4abe-9c46-41be-9d8b-ae68e88ce293-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzmc6\" (UID: \"627d4abe-9c46-41be-9d8b-ae68e88ce293\") " pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:43.428230 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:43.428199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tzmc6" Sep 06 05:58:43.694318 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:43.694288 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tzmc6"] Sep 06 05:58:43.697978 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:43.697949 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627d4abe_9c46_41be_9d8b_ae68e88ce293.slice/crio-8af2b21cfa325eaee5df3028a02e1b15e25f1184e5a1eeb306954f035fffc5ea WatchSource:0}: Error finding container 8af2b21cfa325eaee5df3028a02e1b15e25f1184e5a1eeb306954f035fffc5ea: Status 404 returned error can't find the container with id 8af2b21cfa325eaee5df3028a02e1b15e25f1184e5a1eeb306954f035fffc5ea Sep 06 05:58:44.700845 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:44.700791 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tzmc6" event={"ID":"627d4abe-9c46-41be-9d8b-ae68e88ce293","Type":"ContainerStarted","Data":"99c7eabe0867548446d0142e35045115ea84b5eadc919f9d39c434e4a8dfe789"} Sep 06 05:58:44.701235 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:44.700873 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tzmc6" event={"ID":"627d4abe-9c46-41be-9d8b-ae68e88ce293","Type":"ContainerStarted","Data":"749a2cc359dca0e4e660b635bb71015f37ae763e64f293f1bc2437c8be73d287"} Sep 06 05:58:44.701235 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:44.700894 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tzmc6" event={"ID":"627d4abe-9c46-41be-9d8b-ae68e88ce293","Type":"ContainerStarted","Data":"8af2b21cfa325eaee5df3028a02e1b15e25f1184e5a1eeb306954f035fffc5ea"} Sep 06 05:58:46.709107 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:46.709064 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tzmc6" event={"ID":"627d4abe-9c46-41be-9d8b-ae68e88ce293","Type":"ContainerStarted","Data":"50d69176980aa25b2ad1d204235ea1c05a4a57f3d07114a46c3c9163bf8a03ed"} Sep 06 05:58:46.726426 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:46.726378 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tzmc6" podStartSLOduration=17.570811673 podStartE2EDuration="19.726364682s" totalImagesPullingTime="2.155553009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:58:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:46.726129986 +0000 UTC m=+292.374749164" watchObservedRunningTime="2026-09-06 05:58:46.726364682 +0000 UTC m=+292.374983860" Sep 06 05:58:47.343474 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:47.343434 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6"] Sep 06 05:58:47.346483 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:47.346462 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6" Sep 06 05:58:47.349993 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:47.349969 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 06 05:58:47.350100 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:47.350018 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nbrbh\"" Sep 06 05:58:47.356672 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:47.356650 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6"] Sep 06 05:58:47.519245 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:47.519209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e171e326-abfb-452f-929c-14d0d29e9ffa-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-tsvv6\" (UID: \"e171e326-abfb-452f-929c-14d0d29e9ffa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6" Sep 06 05:58:47.619770 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:47.619683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e171e326-abfb-452f-929c-14d0d29e9ffa-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-tsvv6\" (UID: \"e171e326-abfb-452f-929c-14d0d29e9ffa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6" Sep 06 05:58:47.622130 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:47.622100 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e171e326-abfb-452f-929c-14d0d29e9ffa-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-tsvv6\" (UID: \"e171e326-abfb-452f-929c-14d0d29e9ffa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6" Sep 06 05:58:47.654892 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:47.654859 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6" Sep 06 05:58:47.918964 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:47.918933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6"] Sep 06 05:58:47.921891 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:47.921864 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode171e326_abfb_452f_929c_14d0d29e9ffa.slice/crio-3045a1c437ef401b30e61c2779a16ee058284bacfd80f3ef2d3890d3cafec7a3 WatchSource:0}: Error finding container 3045a1c437ef401b30e61c2779a16ee058284bacfd80f3ef2d3890d3cafec7a3: Status 404 returned error can't find the container with id 3045a1c437ef401b30e61c2779a16ee058284bacfd80f3ef2d3890d3cafec7a3 Sep 06 05:58:48.718329 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:48.718290 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6" event={"ID":"e171e326-abfb-452f-929c-14d0d29e9ffa","Type":"ContainerStarted","Data":"3045a1c437ef401b30e61c2779a16ee058284bacfd80f3ef2d3890d3cafec7a3"} Sep 06 05:58:48.869527 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:48.869482 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2z6qh" podUID="e5ada9e9-c4df-486d-9325-b5dcecdd451b" Sep 06 05:58:49.724743 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:49.724706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6" event={"ID":"e171e326-abfb-452f-929c-14d0d29e9ffa","Type":"ContainerStarted","Data":"0814a0365ced784d70615a783a7cc3d0f12695785eefc432a0d2dbe81e1bea23"} Sep 06 05:58:49.725164 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:49.724866 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6" Sep 06 05:58:49.729477 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:49.729454 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6" Sep 06 05:58:49.760274 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:49.760233 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-tsvv6" podStartSLOduration=1.664887351 podStartE2EDuration="2.76022182s" totalImagesPullingTime="1.095334469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:49.760065449 +0000 UTC m=+295.408684628" watchObservedRunningTime="2026-09-06 05:58:49.76022182 +0000 UTC m=+295.408840999" Sep 06 05:58:52.698734 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:52.698704 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5g9s6" Sep 06 05:58:54.542809 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.542776 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-czz49"] Sep 06 05:58:54.547898 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.547878 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:54.550006 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.549981 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-qq47v"] Sep 06 05:58:54.551763 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.551746 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 06 05:58:54.551883 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.551779 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rcq8p\"" Sep 06 05:58:54.551954 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.551747 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 06 05:58:54.552143 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.552124 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 06 05:58:54.554457 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.554439 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.556668 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.556647 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-czz49"] Sep 06 05:58:54.557789 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.557770 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 06 05:58:54.557946 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.557923 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 06 05:58:54.558012 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.557934 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9j5d8\"" Sep 06 05:58:54.558205 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.558182 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 06 05:58:54.567140 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.567065 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9hfg6"] Sep 06 05:58:54.571648 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.571624 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-qq47v"] Sep 06 05:58:54.571765 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.571744 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.574181 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.574156 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 06 05:58:54.574315 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.574293 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 06 05:58:54.574415 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.574274 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 06 05:58:54.574415 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.574257 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pdq8f\"" Sep 06 05:58:54.675225 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.675374 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.675374 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:54.675374 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.675374 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.675533 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.675533 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675435 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-root\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.675533 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-textfile\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.675533 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-wtmp\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.675533 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.675704 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-sys\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.675704 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.675704 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvtsz\" (UniqueName: \"kubernetes.io/projected/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-kube-api-access-wvtsz\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.675704 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675634 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:54.675704 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675658 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-metrics-client-ca\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.675901 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675696 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl86v\" (UniqueName: \"kubernetes.io/projected/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-kube-api-access-cl86v\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.675901 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675752 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:54.675901 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2dvb\" (UniqueName: \"kubernetes.io/projected/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-kube-api-access-n2dvb\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:54.675901 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.675791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-tls\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.776218 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776183 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.776403 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.776403 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:54.776403 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.776403 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.776403 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-root\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-textfile\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-wtmp\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-sys\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvtsz\" (UniqueName: \"kubernetes.io/projected/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-kube-api-access-wvtsz\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776560 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-metrics-client-ca\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl86v\" (UniqueName: \"kubernetes.io/projected/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-kube-api-access-cl86v\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2dvb\" (UniqueName: \"kubernetes.io/projected/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-kube-api-access-n2dvb\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:54.776660 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.776647 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-tls\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.777296 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.777013 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-root\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.777296 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.777121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.777394 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.777320 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-textfile\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.778940 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.777443 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-wtmp\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.779678 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.779654 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 06 05:58:54.779814 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.779738 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 06 05:58:54.780272 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.780108 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 06 05:58:54.780272 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.780106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-sys\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.780419 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.780286 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 06 05:58:54.783249 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.782987 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 06 05:58:54.783249 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.783014 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 06 05:58:54.783395 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.783264 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 06 05:58:54.783483 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.783467 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 06 05:58:54.783721 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.783699 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 06 05:58:54.789382 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:54.788761 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 06 05:58:54.789382 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:54.788847 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-tls podName:fd6e43a0-e309-4afd-855b-f9ac0d5cec2b nodeName:}" failed. No retries permitted until 2026-09-06 05:58:55.288805079 +0000 UTC m=+300.937424252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-tls") pod "node-exporter-9hfg6" (UID: "fd6e43a0-e309-4afd-855b-f9ac0d5cec2b") : secret "node-exporter-tls" not found Sep 06 05:58:54.790539 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.790518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl86v\" (UniqueName: \"kubernetes.io/projected/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-kube-api-access-cl86v\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.797711 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.793052 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:54.797711 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.793091 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvtsz\" (UniqueName: \"kubernetes.io/projected/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-kube-api-access-wvtsz\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.797711 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.793930 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.797711 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.793989 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:54.797711 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.794470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.797711 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.794895 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.797711 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.797008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.797711 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:54.797396 2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 06 05:58:54.797711 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:58:54.797450 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-openshift-state-metrics-tls podName:33ed7158-81bb-42b5-9afa-7e6cbf9e278a nodeName:}" failed. No retries permitted until 2026-09-06 05:58:55.297432713 +0000 UTC m=+300.946051870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-czz49" (UID: "33ed7158-81bb-42b5-9afa-7e6cbf9e278a") : secret "openshift-state-metrics-tls" not found Sep 06 05:58:54.797711 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.797474 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5e8bcb1-6f79-4e6c-b866-94163a95fd7f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-qq47v\" (UID: \"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:54.798715 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.798069 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-metrics-client-ca\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.800815 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.800790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2dvb\" (UniqueName: \"kubernetes.io/projected/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-kube-api-access-n2dvb\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:54.803099 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.802925 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:54.803869 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.803852 2580 kubelet.go:1648] "Image garbage collection succeeded" Sep 06 05:58:54.872176 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.872049 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9j5d8\"" Sep 06 05:58:54.883642 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:54.879293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" Sep 06 05:58:55.155597 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:55.155570 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-qq47v"] Sep 06 05:58:55.158079 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:55.158045 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e8bcb1_6f79_4e6c_b866_94163a95fd7f.slice/crio-ff716c6ce10c54ba982cb963718d3cfe5fb7e89553c73982b335a18bec9117a4 WatchSource:0}: Error finding container ff716c6ce10c54ba982cb963718d3cfe5fb7e89553c73982b335a18bec9117a4: Status 404 returned error can't find the container with id ff716c6ce10c54ba982cb963718d3cfe5fb7e89553c73982b335a18bec9117a4 Sep 06 05:58:55.382459 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:55.382422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:55.382616 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:55.382467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-tls\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:55.384895 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:55.384870 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd6e43a0-e309-4afd-855b-f9ac0d5cec2b-node-exporter-tls\") pod \"node-exporter-9hfg6\" (UID: \"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b\") " pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:55.385001 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:55.384920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/33ed7158-81bb-42b5-9afa-7e6cbf9e278a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-czz49\" (UID: \"33ed7158-81bb-42b5-9afa-7e6cbf9e278a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:55.464102 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:55.464075 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rcq8p\"" Sep 06 05:58:55.472056 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:55.472033 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" Sep 06 05:58:55.490019 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:55.489990 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pdq8f\"" Sep 06 05:58:55.498180 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:55.498135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9hfg6" Sep 06 05:58:55.510483 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:55.510452 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6e43a0_e309_4afd_855b_f9ac0d5cec2b.slice/crio-8867642c1efc6e080e2ae73a4d375c19eef79fc073cc097947bf8e7ba32e0271 WatchSource:0}: Error finding container 8867642c1efc6e080e2ae73a4d375c19eef79fc073cc097947bf8e7ba32e0271: Status 404 returned error can't find the container with id 8867642c1efc6e080e2ae73a4d375c19eef79fc073cc097947bf8e7ba32e0271 Sep 06 05:58:55.746690 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:55.746527 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hfg6" event={"ID":"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b","Type":"ContainerStarted","Data":"8867642c1efc6e080e2ae73a4d375c19eef79fc073cc097947bf8e7ba32e0271"} Sep 06 05:58:55.747979 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:55.747943 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" event={"ID":"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f","Type":"ContainerStarted","Data":"ff716c6ce10c54ba982cb963718d3cfe5fb7e89553c73982b335a18bec9117a4"} Sep 06 05:58:55.758605 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:55.753730 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-czz49"] Sep 06 05:58:56.530025 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.529989 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr"] Sep 06 05:58:56.534341 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.534311 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.536969 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.536944 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 06 05:58:56.536969 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.536941 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 06 05:58:56.537133 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.537036 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 06 05:58:56.537211 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.537185 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 06 05:58:56.537325 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.537294 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 06 05:58:56.537668 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.537621 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hnh97\"" Sep 06 05:58:56.538010 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.537990 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c0m2cbpdb3a58\"" Sep 06 05:58:56.543439 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.543354 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr"] Sep 06 05:58:56.692814 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.692786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.692943 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.692844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zgj2\" (UniqueName: \"kubernetes.io/projected/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-kube-api-access-5zgj2\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.692943 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.692904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.692943 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.692928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-metrics-client-ca\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.693066 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.692954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-tls\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.693066 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.692982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.693066 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.693019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-grpc-tls\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.693066 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.693054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.753657 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.753614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" event={"ID":"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f","Type":"ContainerStarted","Data":"b2a6bfe6d1d56758d2fe59c8768016f519d0ac5f1dfb528ff8e5587ea73b4f79"} Sep 06 05:58:56.755490 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.755453 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hfg6" event={"ID":"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b","Type":"ContainerStarted","Data":"82a499eeeb3e2bb95dc71d2ff1d77a1dcca64c1b9ea16b1c4f188e3490d6c7bf"} Sep 06 05:58:56.757982 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.757958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" event={"ID":"33ed7158-81bb-42b5-9afa-7e6cbf9e278a","Type":"ContainerStarted","Data":"b4a4ddfa7420453dbc0ca2b90bf81cda0b9604bc6c730f534d57469fa95b5cbd"} Sep 06 05:58:56.758061 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.758014 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" event={"ID":"33ed7158-81bb-42b5-9afa-7e6cbf9e278a","Type":"ContainerStarted","Data":"ba24ba7e71dcc00bcd47480a69c81068c10577f9ced3a318a37a657809dca7e0"} Sep 06 05:58:56.758061 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.758046 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" event={"ID":"33ed7158-81bb-42b5-9afa-7e6cbf9e278a","Type":"ContainerStarted","Data":"f16e50df9b9a709a166c329365ce157cae910a85b70cb3d1bc1b49fb96b8021b"} Sep 06 05:58:56.794472 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.793991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.794472 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.794042 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-metrics-client-ca\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.794472 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.794074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-tls\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.795309 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.795276 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-metrics-client-ca\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.795440 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.795420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.795500 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.795489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-grpc-tls\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.795557 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.795532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.795612 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.795569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.795612 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.795600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zgj2\" (UniqueName: \"kubernetes.io/projected/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-kube-api-access-5zgj2\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.800124 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.800095 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.800409 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.800385 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-grpc-tls\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.800527 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.800506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.800919 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.800896 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-tls\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.802299 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.802247 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.805360 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.805295 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.809879 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.809797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zgj2\" (UniqueName: \"kubernetes.io/projected/2ed6c30a-5c06-4d62-9a24-b5f17515ddf6-kube-api-access-5zgj2\") pod \"thanos-querier-6cb56fbf6c-4n8qr\" (UID: \"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6\") " pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:56.846946 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:56.846870 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:58:57.262681 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:57.262658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr"] Sep 06 05:58:57.264924 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:57.264895 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed6c30a_5c06_4d62_9a24_b5f17515ddf6.slice/crio-168ce925d0422b3b95401f57f07fba631febd5a3820e6d2bcf668236dabae73a WatchSource:0}: Error finding container 168ce925d0422b3b95401f57f07fba631febd5a3820e6d2bcf668236dabae73a: Status 404 returned error can't find the container with id 168ce925d0422b3b95401f57f07fba631febd5a3820e6d2bcf668236dabae73a Sep 06 05:58:57.267067 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:57.267043 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 05:58:57.772600 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:57.772563 2580 generic.go:356] "Generic (PLEG): container finished" podID="fd6e43a0-e309-4afd-855b-f9ac0d5cec2b" containerID="82a499eeeb3e2bb95dc71d2ff1d77a1dcca64c1b9ea16b1c4f188e3490d6c7bf" exitCode=0 Sep 06 05:58:57.773078 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:57.772655 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hfg6" event={"ID":"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b","Type":"ContainerDied","Data":"82a499eeeb3e2bb95dc71d2ff1d77a1dcca64c1b9ea16b1c4f188e3490d6c7bf"} Sep 06 05:58:57.775123 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:57.775096 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" event={"ID":"33ed7158-81bb-42b5-9afa-7e6cbf9e278a","Type":"ContainerStarted","Data":"3e50892d3d9cb825a34bdabe08a0b51096a48730ff3cbc8552ff207b3ba53faa"} Sep 06 05:58:57.776305 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:57.776285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" event={"ID":"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6","Type":"ContainerStarted","Data":"168ce925d0422b3b95401f57f07fba631febd5a3820e6d2bcf668236dabae73a"} Sep 06 05:58:57.778540 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:57.778519 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" event={"ID":"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f","Type":"ContainerStarted","Data":"8fc078f5b03fe31efa26d3ea2c37b660af1f0a376513090426df10bf3d738a5f"} Sep 06 05:58:57.778639 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:57.778595 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" event={"ID":"c5e8bcb1-6f79-4e6c-b866-94163a95fd7f","Type":"ContainerStarted","Data":"29261b1d94ecef733b584035a36b816ccf5d893bbf4c390b204ccc4708fe4cb5"} Sep 06 05:58:57.806888 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:57.806850 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-czz49" podStartSLOduration=2.678679713 podStartE2EDuration="3.806815611s" totalImagesPullingTime="1.128135898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:57.805650061 +0000 UTC m=+303.454269244" watchObservedRunningTime="2026-09-06 05:58:57.806815611 +0000 UTC m=+303.455434804" Sep 06 05:58:57.822878 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:57.822748 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qq47v" podStartSLOduration=2.335264719 podStartE2EDuration="3.822729853s" totalImagesPullingTime="1.487465134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:57.822431681 +0000 UTC m=+303.471050859" watchObservedRunningTime="2026-09-06 05:58:57.822729853 +0000 UTC m=+303.471349033" Sep 06 05:58:58.485980 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.485939 2580 patch_prober.go:28] interesting pod/image-registry-854f5cf89c-xtljb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 06 05:58:58.486158 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.486004 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" podUID="1d264052-c7af-404d-9f73-3a5bed5d755d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 05:58:58.785979 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.785888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hfg6" event={"ID":"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b","Type":"ContainerStarted","Data":"756bd67d285c3c75b4f19c10c9366c15a925722f29d3bba10b97665bc292505b"} Sep 06 05:58:58.785979 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.785963 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hfg6" event={"ID":"fd6e43a0-e309-4afd-855b-f9ac0d5cec2b","Type":"ContainerStarted","Data":"142faf4366b45ea7cce89f793bc3927cf38baede1762811c063b63de3c178ae1"} Sep 06 05:58:58.807265 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.807212 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9hfg6" podStartSLOduration=3.590989338 podStartE2EDuration="4.807190998s" totalImagesPullingTime="1.137568719s" totalInitContainerRuntime="78.632941ms" isStatefulPod=true podCreationTimestamp="2026-09-06 05:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:58.806114582 +0000 UTC m=+304.454733760" watchObservedRunningTime="2026-09-06 05:58:58.807190998 +0000 UTC m=+304.455810179" Sep 06 05:58:58.866645 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.866615 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz"] Sep 06 05:58:58.870271 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.870240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:58.873238 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.873154 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 06 05:58:58.873357 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.873283 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 06 05:58:58.873357 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.873323 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-70g9crsdio17\"" Sep 06 05:58:58.873498 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.873363 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nfvvp\"" Sep 06 05:58:58.873498 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.873396 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 06 05:58:58.873737 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.873712 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 06 05:58:58.880531 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.880508 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz"] Sep 06 05:58:58.915207 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.915179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b7403db-7ece-4d47-b995-cd4c4a139c09-metrics-server-audit-profiles\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:58.915341 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.915215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b7403db-7ece-4d47-b995-cd4c4a139c09-audit-log\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:58.915341 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.915263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhl4\" (UniqueName: \"kubernetes.io/projected/5b7403db-7ece-4d47-b995-cd4c4a139c09-kube-api-access-plhl4\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:58.915435 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.915364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b7403db-7ece-4d47-b995-cd4c4a139c09-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:58.915435 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.915411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b7403db-7ece-4d47-b995-cd4c4a139c09-client-ca-bundle\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:58.915534 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.915487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b7403db-7ece-4d47-b995-cd4c4a139c09-secret-metrics-server-client-certs\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:58.915534 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:58.915517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b7403db-7ece-4d47-b995-cd4c4a139c09-secret-metrics-server-tls\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.015962 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.015929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b7403db-7ece-4d47-b995-cd4c4a139c09-client-ca-bundle\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.016116 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.015995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b7403db-7ece-4d47-b995-cd4c4a139c09-secret-metrics-server-client-certs\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.016116 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.016028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b7403db-7ece-4d47-b995-cd4c4a139c09-secret-metrics-server-tls\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.016116 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.016078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b7403db-7ece-4d47-b995-cd4c4a139c09-metrics-server-audit-profiles\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.016116 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.016105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b7403db-7ece-4d47-b995-cd4c4a139c09-audit-log\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.016342 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.016152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhl4\" (UniqueName: \"kubernetes.io/projected/5b7403db-7ece-4d47-b995-cd4c4a139c09-kube-api-access-plhl4\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.016342 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.016208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b7403db-7ece-4d47-b995-cd4c4a139c09-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.016793 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.016763 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b7403db-7ece-4d47-b995-cd4c4a139c09-audit-log\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.017031 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.016983 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b7403db-7ece-4d47-b995-cd4c4a139c09-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.017446 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.017418 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b7403db-7ece-4d47-b995-cd4c4a139c09-metrics-server-audit-profiles\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.018889 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.018841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b7403db-7ece-4d47-b995-cd4c4a139c09-secret-metrics-server-client-certs\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.019019 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.018997 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b7403db-7ece-4d47-b995-cd4c4a139c09-client-ca-bundle\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.019128 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.019057 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b7403db-7ece-4d47-b995-cd4c4a139c09-secret-metrics-server-tls\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.024839 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.024809 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhl4\" (UniqueName: \"kubernetes.io/projected/5b7403db-7ece-4d47-b995-cd4c4a139c09-kube-api-access-plhl4\") pod \"metrics-server-5ffd6c4df4-xjbbz\" (UID: \"5b7403db-7ece-4d47-b995-cd4c4a139c09\") " pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.182077 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.182035 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:58:59.491186 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.491160 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz"] Sep 06 05:58:59.493020 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:58:59.492998 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7403db_7ece_4d47_b995_cd4c4a139c09.slice/crio-86e1c85be19a0ee8dfa3083e94d828fafc8a49514c924dc2ccaf0169ff273ae5 WatchSource:0}: Error finding container 86e1c85be19a0ee8dfa3083e94d828fafc8a49514c924dc2ccaf0169ff273ae5: Status 404 returned error can't find the container with id 86e1c85be19a0ee8dfa3083e94d828fafc8a49514c924dc2ccaf0169ff273ae5 Sep 06 05:58:59.793788 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.793690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" event={"ID":"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6","Type":"ContainerStarted","Data":"8236814f92b677c7eeb4b1627e1d5c7f30850eaa04c5ccceb9fc4e4c1abfb693"} Sep 06 05:58:59.794326 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.793786 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" event={"ID":"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6","Type":"ContainerStarted","Data":"f1e60bc557ed476b0b288cf4c6f20b5fdadef77dd2de5f7f674a18021c51e961"} Sep 06 05:58:59.794326 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.793853 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" event={"ID":"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6","Type":"ContainerStarted","Data":"114becb4f833431be5e9d29f4aa7c01f0c46cf4c10748f0fe8992b011101188e"} Sep 06 05:58:59.795483 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.795163 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" event={"ID":"5b7403db-7ece-4d47-b995-cd4c4a139c09","Type":"ContainerStarted","Data":"86e1c85be19a0ee8dfa3083e94d828fafc8a49514c924dc2ccaf0169ff273ae5"} Sep 06 05:58:59.868591 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:58:59.868564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 05:59:00.763332 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.763294 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 05:59:00.767515 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.767494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.770219 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.770193 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 06 05:59:00.770219 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.770203 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 06 05:59:00.770407 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.770260 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 06 05:59:00.770407 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.770276 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 06 05:59:00.770557 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.770535 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 06 05:59:00.770557 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.770546 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 06 05:59:00.770727 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.770560 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 06 05:59:00.770727 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.770585 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 06 05:59:00.771430 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.771412 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 06 05:59:00.771636 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.771621 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 06 05:59:00.771783 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.771762 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2nt7w\"" Sep 06 05:59:00.771857 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.771769 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 06 05:59:00.771857 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.771819 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 06 05:59:00.772013 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.771995 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 06 05:59:00.772065 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.772036 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-74pes2m1dhhf4\"" Sep 06 05:59:00.775412 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.775393 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 06 05:59:00.786673 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.786648 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 05:59:00.803279 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.803249 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" event={"ID":"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6","Type":"ContainerStarted","Data":"72ca39e6a2411f0d840cf1a72dc402206d89ee94160c3afd3e8bd135a9160fbd"} Sep 06 05:59:00.803670 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.803324 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" event={"ID":"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6","Type":"ContainerStarted","Data":"263719ceeb14d2c02ca27b3043943c6b0559c653956b02ca6ef18ac4f1f1e5e7"} Sep 06 05:59:00.803670 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.803370 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" event={"ID":"2ed6c30a-5c06-4d62-9a24-b5f17515ddf6","Type":"ContainerStarted","Data":"098680397ec9c45e6dce1b40661f1feb7e5c32e035763494c36494a2c4fe4ac6"} Sep 06 05:59:00.803670 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.803478 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:59:00.829615 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.829546 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" podStartSLOduration=1.9000086999999999 podStartE2EDuration="4.82952671s" totalImagesPullingTime="2.92951801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:59:00.828927435 +0000 UTC m=+306.477546614" watchObservedRunningTime="2026-09-06 05:59:00.82952671 +0000 UTC m=+306.478145892" Sep 06 05:59:00.831332 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.831296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.831456 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.831339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6958968e-fd39-4c82-8a99-0fa8b00fc752-config-out\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.831456 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.831409 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.831456 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.831444 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.831596 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.831469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.831596 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.831498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h62zm\" (UniqueName: \"kubernetes.io/projected/6958968e-fd39-4c82-8a99-0fa8b00fc752-kube-api-access-h62zm\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.831596 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.831539 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.831596 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.831577 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.831730 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.831600 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.831730 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.831701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.831797 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.831741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6958968e-fd39-4c82-8a99-0fa8b00fc752-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.832085 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.832043 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.832194 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.832175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.832261 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.832226 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-web-config\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.832261 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.832254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.832360 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.832336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.832422 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.832397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.832468 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.832428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.832518 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.832491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-config\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.832567 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.832517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933129 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6958968e-fd39-4c82-8a99-0fa8b00fc752-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933129 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933363 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933363 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-web-config\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933363 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933363 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933363 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933610 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933610 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-config\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933610 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933610 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933610 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6958968e-fd39-4c82-8a99-0fa8b00fc752-config-out\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933610 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933610 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933554 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933610 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933555 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933610 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.933610 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h62zm\" (UniqueName: \"kubernetes.io/projected/6958968e-fd39-4c82-8a99-0fa8b00fc752-kube-api-access-h62zm\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.934117 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.934117 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.934117 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.934117 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.933722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.934117 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.934020 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.934117 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.934104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.935208 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.935104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.936941 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.936806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.937443 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.937418 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.937601 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.937548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.937736 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.937716 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.938193 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.937913 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.938397 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.938343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.939032 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.938960 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.939032 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.938990 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.939304 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.939262 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.939633 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.939567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-web-config\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.939754 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.939734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6958968e-fd39-4c82-8a99-0fa8b00fc752-config-out\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.940273 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.940232 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6958968e-fd39-4c82-8a99-0fa8b00fc752-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.940527 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.940505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.940527 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.940515 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.940730 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.940712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-config\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:00.944613 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:00.944594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h62zm\" (UniqueName: \"kubernetes.io/projected/6958968e-fd39-4c82-8a99-0fa8b00fc752-kube-api-access-h62zm\") pod \"prometheus-k8s-0\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:01.079967 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:01.079899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:01.367661 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:01.367586 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 05:59:01.370446 ip-10-0-142-34 kubenswrapper[2580]: W0906 05:59:01.370423 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6958968e_fd39_4c82_8a99_0fa8b00fc752.slice/crio-08b140b7cc9f6d5beccea6e0ce8bc434fb17e13e7b41334c9c96485c2a8f9b78 WatchSource:0}: Error finding container 08b140b7cc9f6d5beccea6e0ce8bc434fb17e13e7b41334c9c96485c2a8f9b78: Status 404 returned error can't find the container with id 08b140b7cc9f6d5beccea6e0ce8bc434fb17e13e7b41334c9c96485c2a8f9b78 Sep 06 05:59:01.687706 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:01.687680 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:59:01.807940 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:01.807906 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerStarted","Data":"08b140b7cc9f6d5beccea6e0ce8bc434fb17e13e7b41334c9c96485c2a8f9b78"} Sep 06 05:59:01.809278 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:01.809251 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" event={"ID":"5b7403db-7ece-4d47-b995-cd4c4a139c09","Type":"ContainerStarted","Data":"f883f95ab165eb1809c20747199437ab8bd7efef7613c4f8253e9d9288f0571e"} Sep 06 05:59:01.826908 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:01.826853 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" podStartSLOduration=2.227368312 podStartE2EDuration="3.826818986s" totalImagesPullingTime="1.599450674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:59:01.825545129 +0000 UTC m=+307.474164312" watchObservedRunningTime="2026-09-06 05:59:01.826818986 +0000 UTC m=+307.475438166" Sep 06 05:59:02.814055 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:02.814021 2580 generic.go:356] "Generic (PLEG): container finished" podID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerID="e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721" exitCode=0 Sep 06 05:59:02.814499 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:02.814109 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerDied","Data":"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721"} Sep 06 05:59:06.815530 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:06.815501 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6cb56fbf6c-4n8qr" Sep 06 05:59:06.833846 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:06.833800 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerStarted","Data":"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a"} Sep 06 05:59:06.834008 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:06.833887 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerStarted","Data":"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745"} Sep 06 05:59:06.834008 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:06.833935 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerStarted","Data":"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d"} Sep 06 05:59:06.834008 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:06.833972 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerStarted","Data":"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3"} Sep 06 05:59:06.834163 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:06.834009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerStarted","Data":"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b"} Sep 06 05:59:06.834163 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:06.834047 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerStarted","Data":"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda"} Sep 06 05:59:06.866615 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:06.866561 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.758273999 podStartE2EDuration="6.866539379s" totalImagesPullingTime="4.089378064s" totalInitContainerRuntime="18.887316ms" isStatefulPod=false podCreationTimestamp="2026-09-06 05:59:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:59:06.863500016 +0000 UTC m=+312.512119208" watchObservedRunningTime="2026-09-06 05:59:06.866539379 +0000 UTC m=+312.515158559" Sep 06 05:59:09.813675 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:09.813642 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-854f5cf89c-xtljb"] Sep 06 05:59:11.080451 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:11.080416 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 05:59:19.183135 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:19.183102 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:59:19.183542 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:19.183159 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:59:34.833662 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:34.833602 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" podUID="1d264052-c7af-404d-9f73-3a5bed5d755d" containerName="registry" containerID="cri-o://6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f" gracePeriod=30 Sep 06 05:59:35.065189 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.065167 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:59:35.229080 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.229051 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d264052-c7af-404d-9f73-3a5bed5d755d-ca-trust-extracted\") pod \"1d264052-c7af-404d-9f73-3a5bed5d755d\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " Sep 06 05:59:35.229258 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.229090 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-certificates\") pod \"1d264052-c7af-404d-9f73-3a5bed5d755d\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " Sep 06 05:59:35.229258 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.229119 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbn2m\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-kube-api-access-xbn2m\") pod \"1d264052-c7af-404d-9f73-3a5bed5d755d\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " Sep 06 05:59:35.229258 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.229147 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-bound-sa-token\") pod \"1d264052-c7af-404d-9f73-3a5bed5d755d\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " Sep 06 05:59:35.229258 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.229171 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d264052-c7af-404d-9f73-3a5bed5d755d-trusted-ca\") pod \"1d264052-c7af-404d-9f73-3a5bed5d755d\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " Sep 06 05:59:35.229258 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.229196 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") pod \"1d264052-c7af-404d-9f73-3a5bed5d755d\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " Sep 06 05:59:35.229258 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.229223 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d264052-c7af-404d-9f73-3a5bed5d755d-installation-pull-secrets\") pod \"1d264052-c7af-404d-9f73-3a5bed5d755d\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " Sep 06 05:59:35.229557 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.229388 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d264052-c7af-404d-9f73-3a5bed5d755d-image-registry-private-configuration\") pod \"1d264052-c7af-404d-9f73-3a5bed5d755d\" (UID: \"1d264052-c7af-404d-9f73-3a5bed5d755d\") " Sep 06 05:59:35.229621 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.229589 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1d264052-c7af-404d-9f73-3a5bed5d755d" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 05:59:35.229683 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.229624 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d264052-c7af-404d-9f73-3a5bed5d755d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1d264052-c7af-404d-9f73-3a5bed5d755d" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 05:59:35.229775 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.229753 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-certificates\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 05:59:35.229775 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.229776 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d264052-c7af-404d-9f73-3a5bed5d755d-trusted-ca\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 05:59:35.231565 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.231529 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1d264052-c7af-404d-9f73-3a5bed5d755d" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 05:59:35.231793 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.231765 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d264052-c7af-404d-9f73-3a5bed5d755d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1d264052-c7af-404d-9f73-3a5bed5d755d" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 05:59:35.231919 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.231900 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d264052-c7af-404d-9f73-3a5bed5d755d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1d264052-c7af-404d-9f73-3a5bed5d755d" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 05:59:35.231983 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.231957 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1d264052-c7af-404d-9f73-3a5bed5d755d" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 05:59:35.232252 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.232227 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-kube-api-access-xbn2m" (OuterVolumeSpecName: "kube-api-access-xbn2m") pod "1d264052-c7af-404d-9f73-3a5bed5d755d" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d"). InnerVolumeSpecName "kube-api-access-xbn2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 05:59:35.237420 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.237399 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d264052-c7af-404d-9f73-3a5bed5d755d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1d264052-c7af-404d-9f73-3a5bed5d755d" (UID: "1d264052-c7af-404d-9f73-3a5bed5d755d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 05:59:35.330987 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.330963 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d264052-c7af-404d-9f73-3a5bed5d755d-ca-trust-extracted\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 05:59:35.330987 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.330986 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbn2m\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-kube-api-access-xbn2m\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 05:59:35.331129 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.330996 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-bound-sa-token\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 05:59:35.331129 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.331004 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d264052-c7af-404d-9f73-3a5bed5d755d-registry-tls\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 05:59:35.331129 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.331013 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d264052-c7af-404d-9f73-3a5bed5d755d-installation-pull-secrets\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 05:59:35.331129 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.331021 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d264052-c7af-404d-9f73-3a5bed5d755d-image-registry-private-configuration\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 05:59:35.946632 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.946595 2580 generic.go:356] "Generic (PLEG): container finished" podID="1d264052-c7af-404d-9f73-3a5bed5d755d" containerID="6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f" exitCode=0 Sep 06 05:59:35.947021 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.946637 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" Sep 06 05:59:35.947021 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.946678 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" event={"ID":"1d264052-c7af-404d-9f73-3a5bed5d755d","Type":"ContainerDied","Data":"6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f"} Sep 06 05:59:35.947021 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.946725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-854f5cf89c-xtljb" event={"ID":"1d264052-c7af-404d-9f73-3a5bed5d755d","Type":"ContainerDied","Data":"439836ce6bafa928774d811b1b50abb2163feec65849f6e01f10855b0639665d"} Sep 06 05:59:35.947021 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.946750 2580 scope.go:117] "RemoveContainer" containerID="6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f" Sep 06 05:59:35.958166 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.958147 2580 scope.go:117] "RemoveContainer" containerID="6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f" Sep 06 05:59:35.958449 ip-10-0-142-34 kubenswrapper[2580]: E0906 05:59:35.958427 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f\": container with ID starting with 6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f not found: ID does not exist" containerID="6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f" Sep 06 05:59:35.958550 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.958455 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f"} err="failed to get container status \"6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f\": rpc error: code = NotFound desc = could not find container \"6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f\": container with ID starting with 6309fc13db837c6baf76d7665d5182cb56f2df9ce5806281bcede0750d7f448f not found: ID does not exist" Sep 06 05:59:35.968811 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.968788 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-854f5cf89c-xtljb"] Sep 06 05:59:35.972258 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:35.972230 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-854f5cf89c-xtljb"] Sep 06 05:59:36.874163 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:36.874130 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d264052-c7af-404d-9f73-3a5bed5d755d" path="/var/lib/kubelet/pods/1d264052-c7af-404d-9f73-3a5bed5d755d/volumes" Sep 06 05:59:39.187687 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:39.187656 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:59:39.191400 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:39.191374 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5ffd6c4df4-xjbbz" Sep 06 05:59:56.023516 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:56.023477 2580 generic.go:356] "Generic (PLEG): container finished" podID="78a98a4b-8e37-43cb-9441-90ed382a2e1a" containerID="33858713598e1823ef40cff07feae15d48f97cc2383cf7fc84834d43d8b8e153" exitCode=0 Sep 06 05:59:56.023885 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:56.023548 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" event={"ID":"78a98a4b-8e37-43cb-9441-90ed382a2e1a","Type":"ContainerDied","Data":"33858713598e1823ef40cff07feae15d48f97cc2383cf7fc84834d43d8b8e153"} Sep 06 05:59:56.023885 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:56.023860 2580 scope.go:117] "RemoveContainer" containerID="33858713598e1823ef40cff07feae15d48f97cc2383cf7fc84834d43d8b8e153" Sep 06 05:59:57.029362 ip-10-0-142-34 kubenswrapper[2580]: I0906 05:59:57.029324 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zmbl6" event={"ID":"78a98a4b-8e37-43cb-9441-90ed382a2e1a","Type":"ContainerStarted","Data":"498266fc56887779579a201a3fa7e87a10348720e32105f5dc0995883dabe269"} Sep 06 06:00:01.080705 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:01.080675 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:01.099903 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:01.099874 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:02.061112 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:02.061090 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:08.878781 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:08.878745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 06:00:08.881016 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:08.880995 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5ada9e9-c4df-486d-9325-b5dcecdd451b-metrics-certs\") pod \"network-metrics-daemon-2z6qh\" (UID: \"e5ada9e9-c4df-486d-9325-b5dcecdd451b\") " pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 06:00:09.172520 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:09.172494 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9rnw9\"" Sep 06 06:00:09.180719 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:09.180695 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2z6qh" Sep 06 06:00:09.446903 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:09.446722 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2z6qh"] Sep 06 06:00:09.449397 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:00:09.449375 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ada9e9_c4df_486d_9325_b5dcecdd451b.slice/crio-923744a1f6a8227f8c9ac74f60059e67fe2e856ddbb4a9dad75511e7833b1e0e WatchSource:0}: Error finding container 923744a1f6a8227f8c9ac74f60059e67fe2e856ddbb4a9dad75511e7833b1e0e: Status 404 returned error can't find the container with id 923744a1f6a8227f8c9ac74f60059e67fe2e856ddbb4a9dad75511e7833b1e0e Sep 06 06:00:10.077707 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:10.077677 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2z6qh" event={"ID":"e5ada9e9-c4df-486d-9325-b5dcecdd451b","Type":"ContainerStarted","Data":"923744a1f6a8227f8c9ac74f60059e67fe2e856ddbb4a9dad75511e7833b1e0e"} Sep 06 06:00:11.085800 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:11.085764 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2z6qh" event={"ID":"e5ada9e9-c4df-486d-9325-b5dcecdd451b","Type":"ContainerStarted","Data":"3fe85fcd7e9b17299dcdea12d1888f4f1890bd5e4c4a74af0dd9d5c2466d1a57"} Sep 06 06:00:11.086237 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:11.085818 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2z6qh" event={"ID":"e5ada9e9-c4df-486d-9325-b5dcecdd451b","Type":"ContainerStarted","Data":"923fd505d80489131fc31214d71ac4c0450ad77899f4f370757712b9b5c2bb8e"} Sep 06 06:00:11.102003 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:11.101956 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2z6qh" podStartSLOduration=374.944299989 podStartE2EDuration="6m16.101940221s" totalImagesPullingTime="1.157640232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:00:11.099963177 +0000 UTC m=+376.748582359" watchObservedRunningTime="2026-09-06 06:00:11.101940221 +0000 UTC m=+376.750559400" Sep 06 06:00:19.119465 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.119436 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 06:00:19.119891 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.119737 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="prometheus" containerID="cri-o://d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda" gracePeriod=600 Sep 06 06:00:19.119891 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.119790 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="kube-rbac-proxy" containerID="cri-o://815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745" gracePeriod=600 Sep 06 06:00:19.120056 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.119792 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="kube-rbac-proxy-thanos" containerID="cri-o://02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a" gracePeriod=600 Sep 06 06:00:19.120589 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.119811 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="config-reloader" containerID="cri-o://5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b" gracePeriod=600 Sep 06 06:00:19.120650 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.119851 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="kube-rbac-proxy-web" containerID="cri-o://9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d" gracePeriod=600 Sep 06 06:00:19.120710 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.119849 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="thanos-sidecar" containerID="cri-o://6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3" gracePeriod=600 Sep 06 06:00:19.448313 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.448291 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:19.571898 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.571868 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-0\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.571898 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.571902 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-kubelet-serving-ca-bundle\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572089 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.571940 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-metrics-client-ca\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572089 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.571967 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-grpc-tls\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572089 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.571990 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h62zm\" (UniqueName: \"kubernetes.io/projected/6958968e-fd39-4c82-8a99-0fa8b00fc752-kube-api-access-h62zm\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572089 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572024 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6958968e-fd39-4c82-8a99-0fa8b00fc752-tls-assets\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572089 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572050 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-metrics-client-certs\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572094 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-web-config\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572121 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-trusted-ca-bundle\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572153 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-tls\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572179 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6958968e-fd39-4c82-8a99-0fa8b00fc752-config-out\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572206 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-serving-certs-ca-bundle\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572241 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572267 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572297 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-db\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572291 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:19.572329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572324 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-2\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572859 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572356 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:19.572859 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572365 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-thanos-prometheus-http-client-file\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572859 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572422 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-config\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572859 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572452 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-1\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572859 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572484 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-kube-rbac-proxy\") pod \"6958968e-fd39-4c82-8a99-0fa8b00fc752\" (UID: \"6958968e-fd39-4c82-8a99-0fa8b00fc752\") " Sep 06 06:00:19.572859 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572763 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.572859 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.572782 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-metrics-client-ca\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.573507 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.573474 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:19.574911 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.574884 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:19.574911 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.574898 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:19.575163 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.575140 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958968e-fd39-4c82-8a99-0fa8b00fc752-config-out" (OuterVolumeSpecName: "config-out") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:00:19.575225 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.575188 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:19.575291 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.575242 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:19.575609 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.575572 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:19.575790 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.575765 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:19.575889 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.575586 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6958968e-fd39-4c82-8a99-0fa8b00fc752-kube-api-access-h62zm" (OuterVolumeSpecName: "kube-api-access-h62zm") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "kube-api-access-h62zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:00:19.575889 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.575794 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:19.575889 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.575669 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6958968e-fd39-4c82-8a99-0fa8b00fc752-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:00:19.576800 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.576772 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:19.576989 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.576964 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-config" (OuterVolumeSpecName: "config") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:19.577141 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.577119 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:00:19.577402 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.577383 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:19.577589 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.577565 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:19.577799 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.577782 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:19.586819 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.586800 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-web-config" (OuterVolumeSpecName: "web-config") pod "6958968e-fd39-4c82-8a99-0fa8b00fc752" (UID: "6958968e-fd39-4c82-8a99-0fa8b00fc752"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:19.673471 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673441 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6958968e-fd39-4c82-8a99-0fa8b00fc752-config-out\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673471 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673473 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673485 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673495 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673504 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-db\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673513 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673522 2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673531 2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-config\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673539 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673547 2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-kube-rbac-proxy\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673555 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673564 2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-grpc-tls\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673573 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h62zm\" (UniqueName: \"kubernetes.io/projected/6958968e-fd39-4c82-8a99-0fa8b00fc752-kube-api-access-h62zm\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673580 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6958968e-fd39-4c82-8a99-0fa8b00fc752-tls-assets\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673938 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673592 2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-metrics-client-certs\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673938 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673601 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-web-config\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673938 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673608 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6958968e-fd39-4c82-8a99-0fa8b00fc752-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:19.673938 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:19.673616 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6958968e-fd39-4c82-8a99-0fa8b00fc752-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:00:20.124275 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124197 2580 generic.go:356] "Generic (PLEG): container finished" podID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerID="02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a" exitCode=0 Sep 06 06:00:20.124275 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124225 2580 generic.go:356] "Generic (PLEG): container finished" podID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerID="815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745" exitCode=0 Sep 06 06:00:20.124275 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124231 2580 generic.go:356] "Generic (PLEG): container finished" podID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerID="9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d" exitCode=0 Sep 06 06:00:20.124275 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124237 2580 generic.go:356] "Generic (PLEG): container finished" podID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerID="6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3" exitCode=0 Sep 06 06:00:20.124275 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124242 2580 generic.go:356] "Generic (PLEG): container finished" podID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerID="5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b" exitCode=0 Sep 06 06:00:20.124275 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124248 2580 generic.go:356] "Generic (PLEG): container finished" podID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerID="d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda" exitCode=0 Sep 06 06:00:20.124275 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124261 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.124926 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124280 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerDied","Data":"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a"} Sep 06 06:00:20.124926 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124356 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerDied","Data":"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745"} Sep 06 06:00:20.124926 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124386 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerDied","Data":"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d"} Sep 06 06:00:20.124926 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124416 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerDied","Data":"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3"} Sep 06 06:00:20.124926 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124424 2580 scope.go:117] "RemoveContainer" containerID="02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a" Sep 06 06:00:20.124926 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124456 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerDied","Data":"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b"} Sep 06 06:00:20.124926 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerDied","Data":"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda"} Sep 06 06:00:20.124926 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.124535 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6958968e-fd39-4c82-8a99-0fa8b00fc752","Type":"ContainerDied","Data":"08b140b7cc9f6d5beccea6e0ce8bc434fb17e13e7b41334c9c96485c2a8f9b78"} Sep 06 06:00:20.132554 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.132535 2580 scope.go:117] "RemoveContainer" containerID="815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745" Sep 06 06:00:20.147517 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.147501 2580 scope.go:117] "RemoveContainer" containerID="9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d" Sep 06 06:00:20.154318 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.154299 2580 scope.go:117] "RemoveContainer" containerID="6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3" Sep 06 06:00:20.158938 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.158913 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 06:00:20.162112 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.162083 2580 scope.go:117] "RemoveContainer" containerID="5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b" Sep 06 06:00:20.163633 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.163616 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 06:00:20.168541 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.168528 2580 scope.go:117] "RemoveContainer" containerID="d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda" Sep 06 06:00:20.175304 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.175289 2580 scope.go:117] "RemoveContainer" containerID="e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721" Sep 06 06:00:20.181873 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.181854 2580 scope.go:117] "RemoveContainer" containerID="02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a" Sep 06 06:00:20.182121 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:00:20.182103 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": container with ID starting with 02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a not found: ID does not exist" containerID="02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a" Sep 06 06:00:20.182181 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.182130 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a"} err="failed to get container status \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": rpc error: code = NotFound desc = could not find container \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": container with ID starting with 02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a not found: ID does not exist" Sep 06 06:00:20.182181 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.182146 2580 scope.go:117] "RemoveContainer" containerID="815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745" Sep 06 06:00:20.182380 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:00:20.182364 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": container with ID starting with 815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745 not found: ID does not exist" containerID="815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745" Sep 06 06:00:20.182420 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.182386 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745"} err="failed to get container status \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": rpc error: code = NotFound desc = could not find container \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": container with ID starting with 815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745 not found: ID does not exist" Sep 06 06:00:20.182420 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.182400 2580 scope.go:117] "RemoveContainer" containerID="9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d" Sep 06 06:00:20.182611 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:00:20.182593 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": container with ID starting with 9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d not found: ID does not exist" containerID="9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d" Sep 06 06:00:20.182681 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.182620 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d"} err="failed to get container status \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": rpc error: code = NotFound desc = could not find container \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": container with ID starting with 9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d not found: ID does not exist" Sep 06 06:00:20.182681 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.182642 2580 scope.go:117] "RemoveContainer" containerID="6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3" Sep 06 06:00:20.182889 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:00:20.182875 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": container with ID starting with 6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3 not found: ID does not exist" containerID="6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3" Sep 06 06:00:20.182933 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.182892 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3"} err="failed to get container status \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": rpc error: code = NotFound desc = could not find container \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": container with ID starting with 6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3 not found: ID does not exist" Sep 06 06:00:20.182933 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.182904 2580 scope.go:117] "RemoveContainer" containerID="5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b" Sep 06 06:00:20.183091 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:00:20.183077 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": container with ID starting with 5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b not found: ID does not exist" containerID="5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b" Sep 06 06:00:20.183124 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.183093 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b"} err="failed to get container status \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": rpc error: code = NotFound desc = could not find container \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": container with ID starting with 5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b not found: ID does not exist" Sep 06 06:00:20.183124 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.183105 2580 scope.go:117] "RemoveContainer" containerID="d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda" Sep 06 06:00:20.183299 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:00:20.183283 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": container with ID starting with d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda not found: ID does not exist" containerID="d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda" Sep 06 06:00:20.183333 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.183303 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda"} err="failed to get container status \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": rpc error: code = NotFound desc = could not find container \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": container with ID starting with d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda not found: ID does not exist" Sep 06 06:00:20.183333 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.183317 2580 scope.go:117] "RemoveContainer" containerID="e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721" Sep 06 06:00:20.183525 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:00:20.183510 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": container with ID starting with e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721 not found: ID does not exist" containerID="e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721" Sep 06 06:00:20.183567 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.183527 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721"} err="failed to get container status \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": rpc error: code = NotFound desc = could not find container \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": container with ID starting with e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721 not found: ID does not exist" Sep 06 06:00:20.183567 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.183539 2580 scope.go:117] "RemoveContainer" containerID="02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a" Sep 06 06:00:20.183722 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.183706 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a"} err="failed to get container status \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": rpc error: code = NotFound desc = could not find container \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": container with ID starting with 02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a not found: ID does not exist" Sep 06 06:00:20.183758 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.183722 2580 scope.go:117] "RemoveContainer" containerID="815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745" Sep 06 06:00:20.183954 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.183938 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745"} err="failed to get container status \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": rpc error: code = NotFound desc = could not find container \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": container with ID starting with 815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745 not found: ID does not exist" Sep 06 06:00:20.184004 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.183954 2580 scope.go:117] "RemoveContainer" containerID="9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d" Sep 06 06:00:20.184160 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.184147 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d"} err="failed to get container status \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": rpc error: code = NotFound desc = could not find container \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": container with ID starting with 9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d not found: ID does not exist" Sep 06 06:00:20.184201 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.184160 2580 scope.go:117] "RemoveContainer" containerID="6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3" Sep 06 06:00:20.184374 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.184354 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3"} err="failed to get container status \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": rpc error: code = NotFound desc = could not find container \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": container with ID starting with 6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3 not found: ID does not exist" Sep 06 06:00:20.184421 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.184375 2580 scope.go:117] "RemoveContainer" containerID="5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b" Sep 06 06:00:20.184571 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.184557 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b"} err="failed to get container status \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": rpc error: code = NotFound desc = could not find container \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": container with ID starting with 5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b not found: ID does not exist" Sep 06 06:00:20.184609 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.184571 2580 scope.go:117] "RemoveContainer" containerID="d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda" Sep 06 06:00:20.184761 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.184746 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda"} err="failed to get container status \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": rpc error: code = NotFound desc = could not find container \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": container with ID starting with d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda not found: ID does not exist" Sep 06 06:00:20.184804 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.184762 2580 scope.go:117] "RemoveContainer" containerID="e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721" Sep 06 06:00:20.184988 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.184972 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721"} err="failed to get container status \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": rpc error: code = NotFound desc = could not find container \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": container with ID starting with e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721 not found: ID does not exist" Sep 06 06:00:20.185036 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.184989 2580 scope.go:117] "RemoveContainer" containerID="02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a" Sep 06 06:00:20.185199 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.185179 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a"} err="failed to get container status \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": rpc error: code = NotFound desc = could not find container \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": container with ID starting with 02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a not found: ID does not exist" Sep 06 06:00:20.185234 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.185199 2580 scope.go:117] "RemoveContainer" containerID="815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745" Sep 06 06:00:20.185391 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.185377 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745"} err="failed to get container status \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": rpc error: code = NotFound desc = could not find container \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": container with ID starting with 815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745 not found: ID does not exist" Sep 06 06:00:20.185432 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.185392 2580 scope.go:117] "RemoveContainer" containerID="9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d" Sep 06 06:00:20.185569 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.185553 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d"} err="failed to get container status \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": rpc error: code = NotFound desc = could not find container \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": container with ID starting with 9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d not found: ID does not exist" Sep 06 06:00:20.185613 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.185569 2580 scope.go:117] "RemoveContainer" containerID="6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3" Sep 06 06:00:20.185752 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.185737 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3"} err="failed to get container status \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": rpc error: code = NotFound desc = could not find container \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": container with ID starting with 6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3 not found: ID does not exist" Sep 06 06:00:20.185792 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.185752 2580 scope.go:117] "RemoveContainer" containerID="5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b" Sep 06 06:00:20.185952 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.185938 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b"} err="failed to get container status \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": rpc error: code = NotFound desc = could not find container \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": container with ID starting with 5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b not found: ID does not exist" Sep 06 06:00:20.185992 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.185952 2580 scope.go:117] "RemoveContainer" containerID="d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda" Sep 06 06:00:20.186157 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.186143 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda"} err="failed to get container status \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": rpc error: code = NotFound desc = could not find container \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": container with ID starting with d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda not found: ID does not exist" Sep 06 06:00:20.186203 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.186158 2580 scope.go:117] "RemoveContainer" containerID="e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721" Sep 06 06:00:20.186343 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.186328 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721"} err="failed to get container status \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": rpc error: code = NotFound desc = could not find container \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": container with ID starting with e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721 not found: ID does not exist" Sep 06 06:00:20.186389 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.186342 2580 scope.go:117] "RemoveContainer" containerID="02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a" Sep 06 06:00:20.186520 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.186505 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a"} err="failed to get container status \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": rpc error: code = NotFound desc = could not find container \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": container with ID starting with 02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a not found: ID does not exist" Sep 06 06:00:20.186561 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.186520 2580 scope.go:117] "RemoveContainer" containerID="815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745" Sep 06 06:00:20.186748 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.186707 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745"} err="failed to get container status \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": rpc error: code = NotFound desc = could not find container \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": container with ID starting with 815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745 not found: ID does not exist" Sep 06 06:00:20.186748 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.186732 2580 scope.go:117] "RemoveContainer" containerID="9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d" Sep 06 06:00:20.187116 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.187002 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d"} err="failed to get container status \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": rpc error: code = NotFound desc = could not find container \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": container with ID starting with 9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d not found: ID does not exist" Sep 06 06:00:20.187116 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.187026 2580 scope.go:117] "RemoveContainer" containerID="6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3" Sep 06 06:00:20.187790 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.187510 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3"} err="failed to get container status \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": rpc error: code = NotFound desc = could not find container \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": container with ID starting with 6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3 not found: ID does not exist" Sep 06 06:00:20.187790 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.187534 2580 scope.go:117] "RemoveContainer" containerID="5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b" Sep 06 06:00:20.187926 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.187788 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b"} err="failed to get container status \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": rpc error: code = NotFound desc = could not find container \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": container with ID starting with 5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b not found: ID does not exist" Sep 06 06:00:20.187926 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.187810 2580 scope.go:117] "RemoveContainer" containerID="d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda" Sep 06 06:00:20.188117 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.188094 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda"} err="failed to get container status \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": rpc error: code = NotFound desc = could not find container \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": container with ID starting with d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda not found: ID does not exist" Sep 06 06:00:20.188159 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.188119 2580 scope.go:117] "RemoveContainer" containerID="e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721" Sep 06 06:00:20.188356 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.188336 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721"} err="failed to get container status \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": rpc error: code = NotFound desc = could not find container \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": container with ID starting with e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721 not found: ID does not exist" Sep 06 06:00:20.188356 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.188356 2580 scope.go:117] "RemoveContainer" containerID="02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a" Sep 06 06:00:20.188618 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.188601 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a"} err="failed to get container status \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": rpc error: code = NotFound desc = could not find container \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": container with ID starting with 02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a not found: ID does not exist" Sep 06 06:00:20.188618 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.188618 2580 scope.go:117] "RemoveContainer" containerID="815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745" Sep 06 06:00:20.188821 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.188802 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745"} err="failed to get container status \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": rpc error: code = NotFound desc = could not find container \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": container with ID starting with 815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745 not found: ID does not exist" Sep 06 06:00:20.188933 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.188821 2580 scope.go:117] "RemoveContainer" containerID="9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d" Sep 06 06:00:20.188987 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.188880 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 06:00:20.189198 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189179 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d"} err="failed to get container status \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": rpc error: code = NotFound desc = could not find container \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": container with ID starting with 9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d not found: ID does not exist" Sep 06 06:00:20.189260 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189200 2580 scope.go:117] "RemoveContainer" containerID="6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3" Sep 06 06:00:20.189260 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189210 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d264052-c7af-404d-9f73-3a5bed5d755d" containerName="registry" Sep 06 06:00:20.189260 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189221 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d264052-c7af-404d-9f73-3a5bed5d755d" containerName="registry" Sep 06 06:00:20.189260 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189232 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="thanos-sidecar" Sep 06 06:00:20.189260 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189238 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="thanos-sidecar" Sep 06 06:00:20.189260 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189250 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="init-config-reloader" Sep 06 06:00:20.189260 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189255 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="init-config-reloader" Sep 06 06:00:20.189260 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189264 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="kube-rbac-proxy-thanos" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189269 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="kube-rbac-proxy-thanos" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189276 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="config-reloader" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189281 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="config-reloader" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189287 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="kube-rbac-proxy" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189293 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="kube-rbac-proxy" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189300 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="prometheus" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189305 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="prometheus" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189313 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="kube-rbac-proxy-web" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189318 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="kube-rbac-proxy-web" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189366 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="kube-rbac-proxy-web" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189377 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="kube-rbac-proxy-thanos" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189387 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d264052-c7af-404d-9f73-3a5bed5d755d" containerName="registry" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189392 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="thanos-sidecar" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189397 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="config-reloader" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189404 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="kube-rbac-proxy" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189411 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" containerName="prometheus" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189418 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3"} err="failed to get container status \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": rpc error: code = NotFound desc = could not find container \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": container with ID starting with 6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3 not found: ID does not exist" Sep 06 06:00:20.189572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189439 2580 scope.go:117] "RemoveContainer" containerID="5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b" Sep 06 06:00:20.190225 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189694 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b"} err="failed to get container status \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": rpc error: code = NotFound desc = could not find container \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": container with ID starting with 5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b not found: ID does not exist" Sep 06 06:00:20.190225 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189710 2580 scope.go:117] "RemoveContainer" containerID="d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda" Sep 06 06:00:20.190225 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189929 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda"} err="failed to get container status \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": rpc error: code = NotFound desc = could not find container \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": container with ID starting with d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda not found: ID does not exist" Sep 06 06:00:20.190225 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.189945 2580 scope.go:117] "RemoveContainer" containerID="e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721" Sep 06 06:00:20.190225 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.190177 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721"} err="failed to get container status \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": rpc error: code = NotFound desc = could not find container \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": container with ID starting with e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721 not found: ID does not exist" Sep 06 06:00:20.190225 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.190194 2580 scope.go:117] "RemoveContainer" containerID="02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a" Sep 06 06:00:20.190434 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.190391 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a"} err="failed to get container status \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": rpc error: code = NotFound desc = could not find container \"02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a\": container with ID starting with 02c273f150f11c9c108102480872ee4b13b16cc4204f796e129ab5c65390d54a not found: ID does not exist" Sep 06 06:00:20.190434 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.190406 2580 scope.go:117] "RemoveContainer" containerID="815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745" Sep 06 06:00:20.190655 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.190635 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745"} err="failed to get container status \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": rpc error: code = NotFound desc = could not find container \"815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745\": container with ID starting with 815f6ec4120c50b34667e20af8227f13df8edcf3cd832ba45fbdcae3f6711745 not found: ID does not exist" Sep 06 06:00:20.190655 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.190655 2580 scope.go:117] "RemoveContainer" containerID="9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d" Sep 06 06:00:20.191026 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.191002 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d"} err="failed to get container status \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": rpc error: code = NotFound desc = could not find container \"9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d\": container with ID starting with 9dc8939cdc28f17288f4ee822fdae693affb2a25133149eb28ee4c63d3edee0d not found: ID does not exist" Sep 06 06:00:20.191026 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.191025 2580 scope.go:117] "RemoveContainer" containerID="6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3" Sep 06 06:00:20.191267 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.191249 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3"} err="failed to get container status \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": rpc error: code = NotFound desc = could not find container \"6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3\": container with ID starting with 6080a3606f22ab2c920d73ceafdf45f958530905ff202fdd2cc4414dc51edff3 not found: ID does not exist" Sep 06 06:00:20.191337 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.191269 2580 scope.go:117] "RemoveContainer" containerID="5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b" Sep 06 06:00:20.191508 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.191488 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b"} err="failed to get container status \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": rpc error: code = NotFound desc = could not find container \"5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b\": container with ID starting with 5dda6e3c94234a6918565b32ceb947a36e013c3b273fe7e9e3f87a5dcb3ca67b not found: ID does not exist" Sep 06 06:00:20.191548 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.191510 2580 scope.go:117] "RemoveContainer" containerID="d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda" Sep 06 06:00:20.191736 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.191715 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda"} err="failed to get container status \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": rpc error: code = NotFound desc = could not find container \"d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda\": container with ID starting with d435c9a694f313df99556d3e17428afacbb30963e880f7fc1541f483ad302fda not found: ID does not exist" Sep 06 06:00:20.191736 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.191734 2580 scope.go:117] "RemoveContainer" containerID="e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721" Sep 06 06:00:20.191991 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.191973 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721"} err="failed to get container status \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": rpc error: code = NotFound desc = could not find container \"e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721\": container with ID starting with e5cd969a3e03fc431b94fa97b0670313f54cecbc0381cd950532ee54fcb76721 not found: ID does not exist" Sep 06 06:00:20.194206 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.194194 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.196604 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.196582 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 06 06:00:20.196679 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.196654 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 06 06:00:20.196679 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.196668 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 06 06:00:20.196774 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.196751 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 06 06:00:20.196994 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.196978 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 06 06:00:20.197111 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.197094 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 06 06:00:20.197382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.197225 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2nt7w\"" Sep 06 06:00:20.197382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.197249 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 06 06:00:20.197382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.197256 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 06 06:00:20.197382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.197261 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 06 06:00:20.197650 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.197639 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-74pes2m1dhhf4\"" Sep 06 06:00:20.197724 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.197704 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 06 06:00:20.197787 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.197736 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 06 06:00:20.198206 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.198187 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 06 06:00:20.199964 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.199945 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 06 06:00:20.207001 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.205802 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 06 06:00:20.209500 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.209481 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 06:00:20.276651 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-config\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.276755 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276658 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.276755 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.276755 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.276755 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276710 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/253382d3-9a00-4927-8088-358f76a673b6-config-out\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.276918 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276761 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.276918 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276804 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.276918 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.276918 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.276918 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.276918 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn9vv\" (UniqueName: \"kubernetes.io/projected/253382d3-9a00-4927-8088-358f76a673b6-kube-api-access-kn9vv\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.276918 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/253382d3-9a00-4927-8088-358f76a673b6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.277120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.277120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.277120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.276994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.277120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.277051 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.277120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.277072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.277120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.277106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-web-config\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.277120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.277121 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/253382d3-9a00-4927-8088-358f76a673b6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.277321 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.277140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.377542 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.377492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.377542 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.377519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.377542 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.377536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-web-config\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.377762 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.377552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/253382d3-9a00-4927-8088-358f76a673b6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.377762 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.377570 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.377762 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.377609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-config\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.377762 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.377630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.377977 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.377896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.377977 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.377935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.377977 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.377963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/253382d3-9a00-4927-8088-358f76a673b6-config-out\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378128 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.377999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378128 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378128 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378049 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378128 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378128 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn9vv\" (UniqueName: \"kubernetes.io/projected/253382d3-9a00-4927-8088-358f76a673b6-kube-api-access-kn9vv\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/253382d3-9a00-4927-8088-358f76a673b6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378271 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378334 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378682 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378682 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378663 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/253382d3-9a00-4927-8088-358f76a673b6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.378971 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.378949 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.380734 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.380707 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.381033 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.381009 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-config\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.381128 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.381105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-web-config\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.381199 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.381143 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.381468 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.381439 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.381914 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.381893 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.382049 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.381954 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.382490 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.382466 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.382735 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.382712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/253382d3-9a00-4927-8088-358f76a673b6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.382950 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.382932 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/253382d3-9a00-4927-8088-358f76a673b6-config-out\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.383309 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.383289 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/253382d3-9a00-4927-8088-358f76a673b6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.383619 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.383602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.383691 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.383661 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.383908 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.383890 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/253382d3-9a00-4927-8088-358f76a673b6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.389500 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.389479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn9vv\" (UniqueName: \"kubernetes.io/projected/253382d3-9a00-4927-8088-358f76a673b6-kube-api-access-kn9vv\") pod \"prometheus-k8s-0\" (UID: \"253382d3-9a00-4927-8088-358f76a673b6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.509997 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.509963 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:20.783849 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.783801 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 06 06:00:20.785669 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:00:20.785641 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253382d3_9a00_4927_8088_358f76a673b6.slice/crio-893ffd2d393fb0387f63ee14f9372e6f5057157fd4b38430f95aca251d7219b0 WatchSource:0}: Error finding container 893ffd2d393fb0387f63ee14f9372e6f5057157fd4b38430f95aca251d7219b0: Status 404 returned error can't find the container with id 893ffd2d393fb0387f63ee14f9372e6f5057157fd4b38430f95aca251d7219b0 Sep 06 06:00:20.874610 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:20.874582 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6958968e-fd39-4c82-8a99-0fa8b00fc752" path="/var/lib/kubelet/pods/6958968e-fd39-4c82-8a99-0fa8b00fc752/volumes" Sep 06 06:00:21.130473 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:21.130397 2580 generic.go:356] "Generic (PLEG): container finished" podID="253382d3-9a00-4927-8088-358f76a673b6" containerID="487c07a0591640a686104ccb4b1abe92b9346dc3c3c6011642f475456816f0e6" exitCode=0 Sep 06 06:00:21.130473 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:21.130441 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"253382d3-9a00-4927-8088-358f76a673b6","Type":"ContainerDied","Data":"487c07a0591640a686104ccb4b1abe92b9346dc3c3c6011642f475456816f0e6"} Sep 06 06:00:21.130993 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:21.130505 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"253382d3-9a00-4927-8088-358f76a673b6","Type":"ContainerStarted","Data":"893ffd2d393fb0387f63ee14f9372e6f5057157fd4b38430f95aca251d7219b0"} Sep 06 06:00:22.139725 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:22.139688 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"253382d3-9a00-4927-8088-358f76a673b6","Type":"ContainerStarted","Data":"fe0a055eee505b358093514459be85a798ccc5ec7d0d62ee59755c8a5313f4d1"} Sep 06 06:00:22.140120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:22.139776 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"253382d3-9a00-4927-8088-358f76a673b6","Type":"ContainerStarted","Data":"a305abe6c2b374b974c8decbf49be42463ca03d1455559e8eada1ec71f3adc96"} Sep 06 06:00:22.140120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:22.139821 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"253382d3-9a00-4927-8088-358f76a673b6","Type":"ContainerStarted","Data":"65693cb7ccacc7ec80f2b2012cfa9a9551ab94da80a280c3e89d61ea273a3fae"} Sep 06 06:00:22.140120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:22.139894 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"253382d3-9a00-4927-8088-358f76a673b6","Type":"ContainerStarted","Data":"4a013dead615d0496550696e30986f4e4c5f5f77dc5eabcade296fd84e2cfb10"} Sep 06 06:00:22.140120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:22.139936 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"253382d3-9a00-4927-8088-358f76a673b6","Type":"ContainerStarted","Data":"660d3d91891c4c1f91dde26f17e1499ffe14d68c0502b75b24c735fde3907ea4"} Sep 06 06:00:22.140120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:22.140001 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"253382d3-9a00-4927-8088-358f76a673b6","Type":"ContainerStarted","Data":"d147787c54d74e79af39e5e434aec6050211489ecdd83fd74abac0aa1aea097f"} Sep 06 06:00:22.165792 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:22.165745 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.145607749 podStartE2EDuration="2.165730132s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.122383ms" isStatefulPod=false podCreationTimestamp="2026-09-06 06:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:00:22.1634854 +0000 UTC m=+387.812104593" watchObservedRunningTime="2026-09-06 06:00:22.165730132 +0000 UTC m=+387.814349311" Sep 06 06:00:25.510724 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:25.510680 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:00:37.644906 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:00:37.644863 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" podUID="d9b61ebb-d415-4305-abd5-d712c4d0ced3" Sep 06 06:00:38.205814 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:38.205778 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 06:00:40.438320 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:40.438280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 06:00:40.440568 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:40.440547 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9b61ebb-d415-4305-abd5-d712c4d0ced3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7dzz\" (UID: \"d9b61ebb-d415-4305-abd5-d712c4d0ced3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 06:00:40.609289 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:40.609259 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nr686\"" Sep 06 06:00:40.616813 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:40.616790 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" Sep 06 06:00:40.886896 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:40.886814 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-f7dzz"] Sep 06 06:00:40.889216 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:00:40.889186 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b61ebb_d415_4305_abd5_d712c4d0ced3.slice/crio-713840ca3efab61457c945739a8268cd36f52f1b52238c846e60c9859bb8933f WatchSource:0}: Error finding container 713840ca3efab61457c945739a8268cd36f52f1b52238c846e60c9859bb8933f: Status 404 returned error can't find the container with id 713840ca3efab61457c945739a8268cd36f52f1b52238c846e60c9859bb8933f Sep 06 06:00:41.218954 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:41.218926 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" event={"ID":"d9b61ebb-d415-4305-abd5-d712c4d0ced3","Type":"ContainerStarted","Data":"713840ca3efab61457c945739a8268cd36f52f1b52238c846e60c9859bb8933f"} Sep 06 06:00:42.224717 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:42.224684 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" event={"ID":"d9b61ebb-d415-4305-abd5-d712c4d0ced3","Type":"ContainerStarted","Data":"f398369e151b659e4eef536cf3758bc04514cc6355e69989818c60a286f5a2a1"} Sep 06 06:00:42.240476 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:00:42.240436 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-f7dzz" podStartSLOduration=378.060259807 podStartE2EDuration="6m19.240420305s" totalImagesPullingTime="1.180160498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:00:42.238684563 +0000 UTC m=+407.887303742" watchObservedRunningTime="2026-09-06 06:00:42.240420305 +0000 UTC m=+407.889039482" Sep 06 06:01:20.510126 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:01:20.510052 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:01:20.525161 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:01:20.525137 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:01:21.394704 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:01:21.394677 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 06 06:03:37.507328 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.507289 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-q26fw"] Sep 06 06:03:37.510503 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.510485 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q26fw" Sep 06 06:03:37.512974 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.512948 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zhmx7\"" Sep 06 06:03:37.514010 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.513992 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 06 06:03:37.514010 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.514000 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 06 06:03:37.521572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.521552 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-q26fw"] Sep 06 06:03:37.585593 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.585557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b024d8c7-3b92-4683-aa51-63896037eeb9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-q26fw\" (UID: \"b024d8c7-3b92-4683-aa51-63896037eeb9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q26fw" Sep 06 06:03:37.585747 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.585623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qpnt\" (UniqueName: \"kubernetes.io/projected/b024d8c7-3b92-4683-aa51-63896037eeb9-kube-api-access-7qpnt\") pod \"cert-manager-cainjector-559cdd8c87-q26fw\" (UID: \"b024d8c7-3b92-4683-aa51-63896037eeb9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q26fw" Sep 06 06:03:37.686460 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.686420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qpnt\" (UniqueName: \"kubernetes.io/projected/b024d8c7-3b92-4683-aa51-63896037eeb9-kube-api-access-7qpnt\") pod \"cert-manager-cainjector-559cdd8c87-q26fw\" (UID: \"b024d8c7-3b92-4683-aa51-63896037eeb9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q26fw" Sep 06 06:03:37.686637 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.686476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b024d8c7-3b92-4683-aa51-63896037eeb9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-q26fw\" (UID: \"b024d8c7-3b92-4683-aa51-63896037eeb9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q26fw" Sep 06 06:03:37.695783 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.695747 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b024d8c7-3b92-4683-aa51-63896037eeb9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-q26fw\" (UID: \"b024d8c7-3b92-4683-aa51-63896037eeb9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q26fw" Sep 06 06:03:37.695897 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.695813 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qpnt\" (UniqueName: \"kubernetes.io/projected/b024d8c7-3b92-4683-aa51-63896037eeb9-kube-api-access-7qpnt\") pod \"cert-manager-cainjector-559cdd8c87-q26fw\" (UID: \"b024d8c7-3b92-4683-aa51-63896037eeb9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q26fw" Sep 06 06:03:37.826582 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:37.826483 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q26fw" Sep 06 06:03:38.094093 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:38.094016 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-q26fw"] Sep 06 06:03:38.097278 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:03:38.097248 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb024d8c7_3b92_4683_aa51_63896037eeb9.slice/crio-eda0bda2e483af46da522a0d9bd92f8fe7d6256000ce926e3cebe9cfb6d256ce WatchSource:0}: Error finding container eda0bda2e483af46da522a0d9bd92f8fe7d6256000ce926e3cebe9cfb6d256ce: Status 404 returned error can't find the container with id eda0bda2e483af46da522a0d9bd92f8fe7d6256000ce926e3cebe9cfb6d256ce Sep 06 06:03:38.938479 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:38.938435 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q26fw" event={"ID":"b024d8c7-3b92-4683-aa51-63896037eeb9","Type":"ContainerStarted","Data":"eda0bda2e483af46da522a0d9bd92f8fe7d6256000ce926e3cebe9cfb6d256ce"} Sep 06 06:03:41.954152 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:41.954105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q26fw" event={"ID":"b024d8c7-3b92-4683-aa51-63896037eeb9","Type":"ContainerStarted","Data":"246e0d69fb3793a6ae75dcca3decd15eca7ca6ac9d73baf9b58017fc59268dea"} Sep 06 06:03:41.970685 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:41.970609 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q26fw" podStartSLOduration=1.732236373 podStartE2EDuration="4.970583367s" totalImagesPullingTime="3.238346994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:03:41.968426991 +0000 UTC m=+587.617046171" watchObservedRunningTime="2026-09-06 06:03:41.970583367 +0000 UTC m=+587.619202545" Sep 06 06:03:57.308311 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.308278 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr"] Sep 06 06:03:57.311402 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.311384 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.315140 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.315111 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 06 06:03:57.315281 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.315144 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 06 06:03:57.315281 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.315116 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 06 06:03:57.315281 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.315113 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 06 06:03:57.315281 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.315278 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 06 06:03:57.315450 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.315396 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4mmn2\"" Sep 06 06:03:57.325860 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.324090 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr"] Sep 06 06:03:57.457501 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.457455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/99f89eb3-6fcb-4940-8d39-a2102c52da51-manager-config\") pod \"lws-controller-manager-6fb67bfd69-hhxwr\" (UID: \"99f89eb3-6fcb-4940-8d39-a2102c52da51\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.457501 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.457513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99f89eb3-6fcb-4940-8d39-a2102c52da51-cert\") pod \"lws-controller-manager-6fb67bfd69-hhxwr\" (UID: \"99f89eb3-6fcb-4940-8d39-a2102c52da51\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.457695 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.457602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qnkg\" (UniqueName: \"kubernetes.io/projected/99f89eb3-6fcb-4940-8d39-a2102c52da51-kube-api-access-2qnkg\") pod \"lws-controller-manager-6fb67bfd69-hhxwr\" (UID: \"99f89eb3-6fcb-4940-8d39-a2102c52da51\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.457695 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.457650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/99f89eb3-6fcb-4940-8d39-a2102c52da51-metrics-cert\") pod \"lws-controller-manager-6fb67bfd69-hhxwr\" (UID: \"99f89eb3-6fcb-4940-8d39-a2102c52da51\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.558710 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.558642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qnkg\" (UniqueName: \"kubernetes.io/projected/99f89eb3-6fcb-4940-8d39-a2102c52da51-kube-api-access-2qnkg\") pod \"lws-controller-manager-6fb67bfd69-hhxwr\" (UID: \"99f89eb3-6fcb-4940-8d39-a2102c52da51\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.558710 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.558682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/99f89eb3-6fcb-4940-8d39-a2102c52da51-metrics-cert\") pod \"lws-controller-manager-6fb67bfd69-hhxwr\" (UID: \"99f89eb3-6fcb-4940-8d39-a2102c52da51\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.558710 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.558710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/99f89eb3-6fcb-4940-8d39-a2102c52da51-manager-config\") pod \"lws-controller-manager-6fb67bfd69-hhxwr\" (UID: \"99f89eb3-6fcb-4940-8d39-a2102c52da51\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.558912 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.558739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99f89eb3-6fcb-4940-8d39-a2102c52da51-cert\") pod \"lws-controller-manager-6fb67bfd69-hhxwr\" (UID: \"99f89eb3-6fcb-4940-8d39-a2102c52da51\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.559436 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.559410 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/99f89eb3-6fcb-4940-8d39-a2102c52da51-manager-config\") pod \"lws-controller-manager-6fb67bfd69-hhxwr\" (UID: \"99f89eb3-6fcb-4940-8d39-a2102c52da51\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.561221 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.561196 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99f89eb3-6fcb-4940-8d39-a2102c52da51-cert\") pod \"lws-controller-manager-6fb67bfd69-hhxwr\" (UID: \"99f89eb3-6fcb-4940-8d39-a2102c52da51\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.561307 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.561254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/99f89eb3-6fcb-4940-8d39-a2102c52da51-metrics-cert\") pod \"lws-controller-manager-6fb67bfd69-hhxwr\" (UID: \"99f89eb3-6fcb-4940-8d39-a2102c52da51\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.567982 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.567958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qnkg\" (UniqueName: \"kubernetes.io/projected/99f89eb3-6fcb-4940-8d39-a2102c52da51-kube-api-access-2qnkg\") pod \"lws-controller-manager-6fb67bfd69-hhxwr\" (UID: \"99f89eb3-6fcb-4940-8d39-a2102c52da51\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.620543 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.620513 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:03:57.897768 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.897689 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr"] Sep 06 06:03:57.901342 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:03:57.901309 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f89eb3_6fcb_4940_8d39_a2102c52da51.slice/crio-96dbbaa77706b71ae76967e186d0e87994f0d1f590365e73d3444ab2ab82de86 WatchSource:0}: Error finding container 96dbbaa77706b71ae76967e186d0e87994f0d1f590365e73d3444ab2ab82de86: Status 404 returned error can't find the container with id 96dbbaa77706b71ae76967e186d0e87994f0d1f590365e73d3444ab2ab82de86 Sep 06 06:03:57.904020 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:57.903997 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 06:03:58.022939 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:03:58.022902 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" event={"ID":"99f89eb3-6fcb-4940-8d39-a2102c52da51","Type":"ContainerStarted","Data":"96dbbaa77706b71ae76967e186d0e87994f0d1f590365e73d3444ab2ab82de86"} Sep 06 06:04:01.036542 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:01.036513 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" event={"ID":"99f89eb3-6fcb-4940-8d39-a2102c52da51","Type":"ContainerStarted","Data":"e9da260f357675a8723456f0665e766d1ed664e4728e0f0df9166c57a038c7af"} Sep 06 06:04:01.036956 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:01.036567 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:04:01.057922 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:01.057877 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" podStartSLOduration=1.7484776530000001 podStartE2EDuration="4.057857178s" totalImagesPullingTime="2.309379525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:04:01.057779988 +0000 UTC m=+606.706399169" watchObservedRunningTime="2026-09-06 06:04:01.057857178 +0000 UTC m=+606.706476363" Sep 06 06:04:04.139207 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.139174 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw"] Sep 06 06:04:04.141770 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.141749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.144463 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.144439 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 06 06:04:04.144651 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.144628 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 06 06:04:04.144740 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.144667 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cvg9l\"" Sep 06 06:04:04.144740 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.144728 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 06 06:04:04.144991 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.144976 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 06 06:04:04.160034 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.160003 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw"] Sep 06 06:04:04.220183 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.220142 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwp54\" (UniqueName: \"kubernetes.io/projected/1def8cda-27d7-4d4b-9f76-85b083cad401-kube-api-access-hwp54\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.220183 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.220189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1def8cda-27d7-4d4b-9f76-85b083cad401-tmp\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.220395 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.220265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1def8cda-27d7-4d4b-9f76-85b083cad401-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.220395 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.220289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1def8cda-27d7-4d4b-9f76-85b083cad401-webhook-cert\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.220395 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.220316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1def8cda-27d7-4d4b-9f76-85b083cad401-manifests\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.320713 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.320677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1def8cda-27d7-4d4b-9f76-85b083cad401-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.320713 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.320719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1def8cda-27d7-4d4b-9f76-85b083cad401-webhook-cert\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.320960 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.320843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1def8cda-27d7-4d4b-9f76-85b083cad401-manifests\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.320960 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.320906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwp54\" (UniqueName: \"kubernetes.io/projected/1def8cda-27d7-4d4b-9f76-85b083cad401-kube-api-access-hwp54\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.320960 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.320937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1def8cda-27d7-4d4b-9f76-85b083cad401-tmp\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.321233 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.321214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1def8cda-27d7-4d4b-9f76-85b083cad401-manifests\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.323099 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.323076 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1def8cda-27d7-4d4b-9f76-85b083cad401-tmp\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.323302 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.323280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1def8cda-27d7-4d4b-9f76-85b083cad401-webhook-cert\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.323366 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.323353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1def8cda-27d7-4d4b-9f76-85b083cad401-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.328122 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.328096 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwp54\" (UniqueName: \"kubernetes.io/projected/1def8cda-27d7-4d4b-9f76-85b083cad401-kube-api-access-hwp54\") pod \"opendatahub-operator-controller-manager-b897cd959-76mrw\" (UID: \"1def8cda-27d7-4d4b-9f76-85b083cad401\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.452324 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.452286 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:04.727693 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:04.727655 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw"] Sep 06 06:04:04.730433 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:04:04.730407 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1def8cda_27d7_4d4b_9f76_85b083cad401.slice/crio-869ccb5f6cf10db38f5ecb97a48072157bd16442c4577ea626e8470abb71eab4 WatchSource:0}: Error finding container 869ccb5f6cf10db38f5ecb97a48072157bd16442c4577ea626e8470abb71eab4: Status 404 returned error can't find the container with id 869ccb5f6cf10db38f5ecb97a48072157bd16442c4577ea626e8470abb71eab4 Sep 06 06:04:05.054399 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:05.054317 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" event={"ID":"1def8cda-27d7-4d4b-9f76-85b083cad401","Type":"ContainerStarted","Data":"869ccb5f6cf10db38f5ecb97a48072157bd16442c4577ea626e8470abb71eab4"} Sep 06 06:04:08.072944 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:08.072901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" event={"ID":"1def8cda-27d7-4d4b-9f76-85b083cad401","Type":"ContainerStarted","Data":"ee37e5eee81e51f4f3f22d350352cb89a99ff81ef6302428668f3929856c5267"} Sep 06 06:04:09.078311 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:09.078218 2580 generic.go:356] "Generic (PLEG): container finished" podID="1def8cda-27d7-4d4b-9f76-85b083cad401" containerID="ee37e5eee81e51f4f3f22d350352cb89a99ff81ef6302428668f3929856c5267" exitCode=0 Sep 06 06:04:09.078640 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:09.078298 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" event={"ID":"1def8cda-27d7-4d4b-9f76-85b083cad401","Type":"ContainerDied","Data":"ee37e5eee81e51f4f3f22d350352cb89a99ff81ef6302428668f3929856c5267"} Sep 06 06:04:10.086016 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:10.085981 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" event={"ID":"1def8cda-27d7-4d4b-9f76-85b083cad401","Type":"ContainerStarted","Data":"bfec2d216de959db0ea44e9e33bd4a38849fc7089790162d29c5e9c46a93826b"} Sep 06 06:04:10.086494 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:10.086067 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:10.108420 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:10.108373 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" podStartSLOduration=1.906172208 podStartE2EDuration="6.108358591s" totalImagesPullingTime="3.172916199s" totalInitContainerRuntime="1.029270184s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:04:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:04:10.106218888 +0000 UTC m=+615.754838068" watchObservedRunningTime="2026-09-06 06:04:10.108358591 +0000 UTC m=+615.756977770" Sep 06 06:04:12.042747 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:12.042719 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-hhxwr" Sep 06 06:04:21.096848 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:21.096759 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-76mrw" Sep 06 06:04:23.314420 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.314378 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-696f56895b-nklb9"] Sep 06 06:04:23.316682 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.316664 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" Sep 06 06:04:23.324650 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.324622 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 06 06:04:23.326010 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.325985 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 06 06:04:23.326194 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.326174 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 06 06:04:23.328115 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.328098 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 06 06:04:23.328276 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.328263 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9m959\"" Sep 06 06:04:23.351779 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.351757 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-696f56895b-nklb9"] Sep 06 06:04:23.376352 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.376332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w2km\" (UniqueName: \"kubernetes.io/projected/b3da34e8-2b57-4e9d-8f2a-fd7067b2c590-kube-api-access-4w2km\") pod \"kube-auth-proxy-696f56895b-nklb9\" (UID: \"b3da34e8-2b57-4e9d-8f2a-fd7067b2c590\") " pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" Sep 06 06:04:23.376440 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.376375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da34e8-2b57-4e9d-8f2a-fd7067b2c590-tls-certs\") pod \"kube-auth-proxy-696f56895b-nklb9\" (UID: \"b3da34e8-2b57-4e9d-8f2a-fd7067b2c590\") " pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" Sep 06 06:04:23.376481 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.376452 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3da34e8-2b57-4e9d-8f2a-fd7067b2c590-tmp\") pod \"kube-auth-proxy-696f56895b-nklb9\" (UID: \"b3da34e8-2b57-4e9d-8f2a-fd7067b2c590\") " pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" Sep 06 06:04:23.476919 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.476889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da34e8-2b57-4e9d-8f2a-fd7067b2c590-tls-certs\") pod \"kube-auth-proxy-696f56895b-nklb9\" (UID: \"b3da34e8-2b57-4e9d-8f2a-fd7067b2c590\") " pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" Sep 06 06:04:23.477033 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.476945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3da34e8-2b57-4e9d-8f2a-fd7067b2c590-tmp\") pod \"kube-auth-proxy-696f56895b-nklb9\" (UID: \"b3da34e8-2b57-4e9d-8f2a-fd7067b2c590\") " pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" Sep 06 06:04:23.477033 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.476976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w2km\" (UniqueName: \"kubernetes.io/projected/b3da34e8-2b57-4e9d-8f2a-fd7067b2c590-kube-api-access-4w2km\") pod \"kube-auth-proxy-696f56895b-nklb9\" (UID: \"b3da34e8-2b57-4e9d-8f2a-fd7067b2c590\") " pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" Sep 06 06:04:23.479180 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.479160 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3da34e8-2b57-4e9d-8f2a-fd7067b2c590-tmp\") pod \"kube-auth-proxy-696f56895b-nklb9\" (UID: \"b3da34e8-2b57-4e9d-8f2a-fd7067b2c590\") " pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" Sep 06 06:04:23.479382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.479363 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da34e8-2b57-4e9d-8f2a-fd7067b2c590-tls-certs\") pod \"kube-auth-proxy-696f56895b-nklb9\" (UID: \"b3da34e8-2b57-4e9d-8f2a-fd7067b2c590\") " pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" Sep 06 06:04:23.495204 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.495175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w2km\" (UniqueName: \"kubernetes.io/projected/b3da34e8-2b57-4e9d-8f2a-fd7067b2c590-kube-api-access-4w2km\") pod \"kube-auth-proxy-696f56895b-nklb9\" (UID: \"b3da34e8-2b57-4e9d-8f2a-fd7067b2c590\") " pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" Sep 06 06:04:23.631376 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.631320 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" Sep 06 06:04:23.906991 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:23.906964 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-696f56895b-nklb9"] Sep 06 06:04:23.908997 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:04:23.908972 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3da34e8_2b57_4e9d_8f2a_fd7067b2c590.slice/crio-b65a20f0e95be661999543a28593008d9f1be21bff50b3b943665d2350369dae WatchSource:0}: Error finding container b65a20f0e95be661999543a28593008d9f1be21bff50b3b943665d2350369dae: Status 404 returned error can't find the container with id b65a20f0e95be661999543a28593008d9f1be21bff50b3b943665d2350369dae Sep 06 06:04:24.152138 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:24.152088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" event={"ID":"b3da34e8-2b57-4e9d-8f2a-fd7067b2c590","Type":"ContainerStarted","Data":"b65a20f0e95be661999543a28593008d9f1be21bff50b3b943665d2350369dae"} Sep 06 06:04:27.170080 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:27.170051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" event={"ID":"b3da34e8-2b57-4e9d-8f2a-fd7067b2c590","Type":"ContainerStarted","Data":"18bb9f4eb106d909f71fd546ec854fb76781739bcf5805cb1d529ec85a12fc96"} Sep 06 06:04:27.185145 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:04:27.185090 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-696f56895b-nklb9" podStartSLOduration=1.480085274 podStartE2EDuration="4.185071607s" totalImagesPullingTime="2.704986333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:04:27.184912253 +0000 UTC m=+632.833531436" watchObservedRunningTime="2026-09-06 06:04:27.185071607 +0000 UTC m=+632.833690787" Sep 06 06:05:55.764149 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:55.764072 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-n875r"] Sep 06 06:05:55.767336 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:55.767318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-n875r" Sep 06 06:05:55.770076 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:55.770046 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 06 06:05:55.770249 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:55.770159 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 06 06:05:55.770249 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:55.770193 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cj4lg\"" Sep 06 06:05:55.770409 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:55.770247 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 06 06:05:55.779818 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:55.779794 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-n875r"] Sep 06 06:05:55.857911 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:55.857864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h4vz\" (UniqueName: \"kubernetes.io/projected/dcf33542-3d05-4fb3-a4e8-76ac2142fdd7-kube-api-access-7h4vz\") pod \"dns-operator-controller-manager-5985564776-n875r\" (UID: \"dcf33542-3d05-4fb3-a4e8-76ac2142fdd7\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-n875r" Sep 06 06:05:55.958882 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:55.958849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4vz\" (UniqueName: \"kubernetes.io/projected/dcf33542-3d05-4fb3-a4e8-76ac2142fdd7-kube-api-access-7h4vz\") pod \"dns-operator-controller-manager-5985564776-n875r\" (UID: \"dcf33542-3d05-4fb3-a4e8-76ac2142fdd7\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-n875r" Sep 06 06:05:55.968217 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:55.968194 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4vz\" (UniqueName: \"kubernetes.io/projected/dcf33542-3d05-4fb3-a4e8-76ac2142fdd7-kube-api-access-7h4vz\") pod \"dns-operator-controller-manager-5985564776-n875r\" (UID: \"dcf33542-3d05-4fb3-a4e8-76ac2142fdd7\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-n875r" Sep 06 06:05:56.077434 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:56.077350 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-n875r" Sep 06 06:05:56.351799 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:56.351772 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-n875r"] Sep 06 06:05:56.354970 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:05:56.354925 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf33542_3d05_4fb3_a4e8_76ac2142fdd7.slice/crio-347b61fd8359d2b3546272f2049dd0dda245a6506fde07b7180d8b7c1e415597 WatchSource:0}: Error finding container 347b61fd8359d2b3546272f2049dd0dda245a6506fde07b7180d8b7c1e415597: Status 404 returned error can't find the container with id 347b61fd8359d2b3546272f2049dd0dda245a6506fde07b7180d8b7c1e415597 Sep 06 06:05:56.578364 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:56.578328 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-n875r" event={"ID":"dcf33542-3d05-4fb3-a4e8-76ac2142fdd7","Type":"ContainerStarted","Data":"347b61fd8359d2b3546272f2049dd0dda245a6506fde07b7180d8b7c1e415597"} Sep 06 06:05:58.362692 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:58.362655 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-5gnds"] Sep 06 06:05:58.371939 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:58.371908 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-5gnds" Sep 06 06:05:58.374790 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:58.374762 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ckhnc\"" Sep 06 06:05:58.376354 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:58.376324 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-5gnds"] Sep 06 06:05:58.485176 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:58.485138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6hj9\" (UniqueName: \"kubernetes.io/projected/e4393b5c-d446-4dbc-8070-b12a57dd98a9-kube-api-access-k6hj9\") pod \"authorino-operator-68d88948d8-5gnds\" (UID: \"e4393b5c-d446-4dbc-8070-b12a57dd98a9\") " pod="kuadrant-system/authorino-operator-68d88948d8-5gnds" Sep 06 06:05:58.586500 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:58.586462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6hj9\" (UniqueName: \"kubernetes.io/projected/e4393b5c-d446-4dbc-8070-b12a57dd98a9-kube-api-access-k6hj9\") pod \"authorino-operator-68d88948d8-5gnds\" (UID: \"e4393b5c-d446-4dbc-8070-b12a57dd98a9\") " pod="kuadrant-system/authorino-operator-68d88948d8-5gnds" Sep 06 06:05:58.598506 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:58.598479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6hj9\" (UniqueName: \"kubernetes.io/projected/e4393b5c-d446-4dbc-8070-b12a57dd98a9-kube-api-access-k6hj9\") pod \"authorino-operator-68d88948d8-5gnds\" (UID: \"e4393b5c-d446-4dbc-8070-b12a57dd98a9\") " pod="kuadrant-system/authorino-operator-68d88948d8-5gnds" Sep 06 06:05:58.684645 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:58.684598 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-5gnds" Sep 06 06:05:59.289543 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:59.289507 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-5gnds"] Sep 06 06:05:59.291735 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:05:59.291709 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4393b5c_d446_4dbc_8070_b12a57dd98a9.slice/crio-b1b95eb644916be4dfc87be8e3f73bcc089965d2e40e7ffb94b0dc768d43e5b6 WatchSource:0}: Error finding container b1b95eb644916be4dfc87be8e3f73bcc089965d2e40e7ffb94b0dc768d43e5b6: Status 404 returned error can't find the container with id b1b95eb644916be4dfc87be8e3f73bcc089965d2e40e7ffb94b0dc768d43e5b6 Sep 06 06:05:59.596897 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:59.596788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-5gnds" event={"ID":"e4393b5c-d446-4dbc-8070-b12a57dd98a9","Type":"ContainerStarted","Data":"b1b95eb644916be4dfc87be8e3f73bcc089965d2e40e7ffb94b0dc768d43e5b6"} Sep 06 06:05:59.598508 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:59.598480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-n875r" event={"ID":"dcf33542-3d05-4fb3-a4e8-76ac2142fdd7","Type":"ContainerStarted","Data":"92ddbd2a0b610509e03b78e63f506a8d0e9b5fb418e2de2edd74398195b4373b"} Sep 06 06:05:59.598623 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:59.598557 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-n875r" Sep 06 06:05:59.636492 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:05:59.636426 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-n875r" podStartSLOduration=1.902548889 podStartE2EDuration="4.636411656s" totalImagesPullingTime="2.733862767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:05:59.635576728 +0000 UTC m=+725.284195906" watchObservedRunningTime="2026-09-06 06:05:59.636411656 +0000 UTC m=+725.285030834" Sep 06 06:06:01.610615 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:01.610571 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-5gnds" event={"ID":"e4393b5c-d446-4dbc-8070-b12a57dd98a9","Type":"ContainerStarted","Data":"74fb94ce35a4e55cb8f23d0a2b5ea518fb051b33e651cbd038d8450c094df69e"} Sep 06 06:06:01.611025 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:01.610698 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-5gnds" Sep 06 06:06:01.628945 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:01.628894 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-5gnds" podStartSLOduration=2.357335855 podStartE2EDuration="3.628877277s" totalImagesPullingTime="1.271541422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:06:01.627388826 +0000 UTC m=+727.276008005" watchObservedRunningTime="2026-09-06 06:06:01.628877277 +0000 UTC m=+727.277496456" Sep 06 06:06:10.607164 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:10.607129 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-n875r" Sep 06 06:06:12.617705 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:12.617671 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-5gnds" Sep 06 06:06:44.043866 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.043809 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-2szpp"] Sep 06 06:06:44.048645 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.048622 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-2szpp" Sep 06 06:06:44.051763 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.051734 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tldxr\"" Sep 06 06:06:44.053341 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.053315 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2szpp"] Sep 06 06:06:44.085444 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.085417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47gh\" (UniqueName: \"kubernetes.io/projected/bc2d8c5a-4ece-472d-be3d-2ef60aa987ed-kube-api-access-x47gh\") pod \"authorino-b54df94b4-2szpp\" (UID: \"bc2d8c5a-4ece-472d-be3d-2ef60aa987ed\") " pod="kuadrant-system/authorino-b54df94b4-2szpp" Sep 06 06:06:44.174479 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.174446 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-rcwt7"] Sep 06 06:06:44.177016 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.177001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-rcwt7" Sep 06 06:06:44.181000 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.180974 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-rcwt7"] Sep 06 06:06:44.186009 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.185985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47gh\" (UniqueName: \"kubernetes.io/projected/bc2d8c5a-4ece-472d-be3d-2ef60aa987ed-kube-api-access-x47gh\") pod \"authorino-b54df94b4-2szpp\" (UID: \"bc2d8c5a-4ece-472d-be3d-2ef60aa987ed\") " pod="kuadrant-system/authorino-b54df94b4-2szpp" Sep 06 06:06:44.193814 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.193784 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47gh\" (UniqueName: \"kubernetes.io/projected/bc2d8c5a-4ece-472d-be3d-2ef60aa987ed-kube-api-access-x47gh\") pod \"authorino-b54df94b4-2szpp\" (UID: \"bc2d8c5a-4ece-472d-be3d-2ef60aa987ed\") " pod="kuadrant-system/authorino-b54df94b4-2szpp" Sep 06 06:06:44.287082 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.287055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52hb\" (UniqueName: \"kubernetes.io/projected/91929dda-9d6d-4225-bb28-b75f42fe6dcb-kube-api-access-j52hb\") pod \"authorino-c548b5d7d-rcwt7\" (UID: \"91929dda-9d6d-4225-bb28-b75f42fe6dcb\") " pod="kuadrant-system/authorino-c548b5d7d-rcwt7" Sep 06 06:06:44.358806 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.358739 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-2szpp" Sep 06 06:06:44.388386 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.388356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52hb\" (UniqueName: \"kubernetes.io/projected/91929dda-9d6d-4225-bb28-b75f42fe6dcb-kube-api-access-j52hb\") pod \"authorino-c548b5d7d-rcwt7\" (UID: \"91929dda-9d6d-4225-bb28-b75f42fe6dcb\") " pod="kuadrant-system/authorino-c548b5d7d-rcwt7" Sep 06 06:06:44.397171 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.397143 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52hb\" (UniqueName: \"kubernetes.io/projected/91929dda-9d6d-4225-bb28-b75f42fe6dcb-kube-api-access-j52hb\") pod \"authorino-c548b5d7d-rcwt7\" (UID: \"91929dda-9d6d-4225-bb28-b75f42fe6dcb\") " pod="kuadrant-system/authorino-c548b5d7d-rcwt7" Sep 06 06:06:44.487176 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.487141 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-rcwt7" Sep 06 06:06:44.624922 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.624896 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2szpp"] Sep 06 06:06:44.627030 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:06:44.626996 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc2d8c5a_4ece_472d_be3d_2ef60aa987ed.slice/crio-2bd428c1704a6a6e78ab64bac725f025869b3ea62ee61f1ba6b8a7181af99364 WatchSource:0}: Error finding container 2bd428c1704a6a6e78ab64bac725f025869b3ea62ee61f1ba6b8a7181af99364: Status 404 returned error can't find the container with id 2bd428c1704a6a6e78ab64bac725f025869b3ea62ee61f1ba6b8a7181af99364 Sep 06 06:06:44.751528 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.751502 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-rcwt7"] Sep 06 06:06:44.752914 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:06:44.752890 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91929dda_9d6d_4225_bb28_b75f42fe6dcb.slice/crio-86f4c9966e855a655f242fb8e7d6a06f7521d5283557091064502806a25c2cc5 WatchSource:0}: Error finding container 86f4c9966e855a655f242fb8e7d6a06f7521d5283557091064502806a25c2cc5: Status 404 returned error can't find the container with id 86f4c9966e855a655f242fb8e7d6a06f7521d5283557091064502806a25c2cc5 Sep 06 06:06:44.813397 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.813355 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-2szpp" event={"ID":"bc2d8c5a-4ece-472d-be3d-2ef60aa987ed","Type":"ContainerStarted","Data":"2bd428c1704a6a6e78ab64bac725f025869b3ea62ee61f1ba6b8a7181af99364"} Sep 06 06:06:44.814544 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:44.814518 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-rcwt7" event={"ID":"91929dda-9d6d-4225-bb28-b75f42fe6dcb","Type":"ContainerStarted","Data":"86f4c9966e855a655f242fb8e7d6a06f7521d5283557091064502806a25c2cc5"} Sep 06 06:06:46.837416 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:46.837358 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-rcwt7" event={"ID":"91929dda-9d6d-4225-bb28-b75f42fe6dcb","Type":"ContainerStarted","Data":"7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb"} Sep 06 06:06:46.839386 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:46.839353 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-2szpp" event={"ID":"bc2d8c5a-4ece-472d-be3d-2ef60aa987ed","Type":"ContainerStarted","Data":"3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5"} Sep 06 06:06:46.853598 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:46.853542 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-rcwt7" podStartSLOduration=0.938628561 podStartE2EDuration="2.853526172s" totalImagesPullingTime="1.914897611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:06:46.852240125 +0000 UTC m=+772.500859301" watchObservedRunningTime="2026-09-06 06:06:46.853526172 +0000 UTC m=+772.502145351" Sep 06 06:06:46.868737 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:46.868693 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-2szpp" podStartSLOduration=0.816793175 podStartE2EDuration="2.868670811s" totalImagesPullingTime="2.051877636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:06:46.867541118 +0000 UTC m=+772.516160298" watchObservedRunningTime="2026-09-06 06:06:46.868670811 +0000 UTC m=+772.517289991" Sep 06 06:06:46.893387 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:46.893353 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2szpp"] Sep 06 06:06:48.849171 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:48.849130 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-2szpp" podUID="bc2d8c5a-4ece-472d-be3d-2ef60aa987ed" containerName="authorino" containerID="cri-o://3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5" gracePeriod=30 Sep 06 06:06:49.098421 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.098395 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-2szpp" Sep 06 06:06:49.233506 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.233473 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x47gh\" (UniqueName: \"kubernetes.io/projected/bc2d8c5a-4ece-472d-be3d-2ef60aa987ed-kube-api-access-x47gh\") pod \"bc2d8c5a-4ece-472d-be3d-2ef60aa987ed\" (UID: \"bc2d8c5a-4ece-472d-be3d-2ef60aa987ed\") " Sep 06 06:06:49.235650 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.235626 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2d8c5a-4ece-472d-be3d-2ef60aa987ed-kube-api-access-x47gh" (OuterVolumeSpecName: "kube-api-access-x47gh") pod "bc2d8c5a-4ece-472d-be3d-2ef60aa987ed" (UID: "bc2d8c5a-4ece-472d-be3d-2ef60aa987ed"). InnerVolumeSpecName "kube-api-access-x47gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:06:49.334527 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.334498 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47gh\" (UniqueName: \"kubernetes.io/projected/bc2d8c5a-4ece-472d-be3d-2ef60aa987ed-kube-api-access-x47gh\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:06:49.854632 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.854596 2580 generic.go:356] "Generic (PLEG): container finished" podID="bc2d8c5a-4ece-472d-be3d-2ef60aa987ed" containerID="3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5" exitCode=0 Sep 06 06:06:49.855044 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.854640 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-2szpp" Sep 06 06:06:49.855044 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.854656 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-2szpp" event={"ID":"bc2d8c5a-4ece-472d-be3d-2ef60aa987ed","Type":"ContainerDied","Data":"3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5"} Sep 06 06:06:49.855044 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.854690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-2szpp" event={"ID":"bc2d8c5a-4ece-472d-be3d-2ef60aa987ed","Type":"ContainerDied","Data":"2bd428c1704a6a6e78ab64bac725f025869b3ea62ee61f1ba6b8a7181af99364"} Sep 06 06:06:49.855044 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.854707 2580 scope.go:117] "RemoveContainer" containerID="3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5" Sep 06 06:06:49.863420 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.863394 2580 scope.go:117] "RemoveContainer" containerID="3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5" Sep 06 06:06:49.863719 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:06:49.863698 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5\": container with ID starting with 3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5 not found: ID does not exist" containerID="3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5" Sep 06 06:06:49.863766 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.863729 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5"} err="failed to get container status \"3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5\": rpc error: code = NotFound desc = could not find container \"3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5\": container with ID starting with 3651716e4c469d215e41ed8247c01445ffda2d7dffe224f7c81dff7a78ef68d5 not found: ID does not exist" Sep 06 06:06:49.874609 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.874577 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2szpp"] Sep 06 06:06:49.876561 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:49.876537 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2szpp"] Sep 06 06:06:50.875405 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:06:50.875357 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc2d8c5a-4ece-472d-be3d-2ef60aa987ed" path="/var/lib/kubelet/pods/bc2d8c5a-4ece-472d-be3d-2ef60aa987ed/volumes" Sep 06 06:07:12.365602 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.365563 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9nfmb"] Sep 06 06:07:12.366127 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.365961 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc2d8c5a-4ece-472d-be3d-2ef60aa987ed" containerName="authorino" Sep 06 06:07:12.366127 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.365974 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2d8c5a-4ece-472d-be3d-2ef60aa987ed" containerName="authorino" Sep 06 06:07:12.366127 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.366042 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc2d8c5a-4ece-472d-be3d-2ef60aa987ed" containerName="authorino" Sep 06 06:07:12.370189 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.370169 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-9nfmb" Sep 06 06:07:12.375691 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.375669 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9nfmb"] Sep 06 06:07:12.464905 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.464871 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9nfmb"] Sep 06 06:07:12.465158 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:07:12.465139 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-j5zzq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-9nfmb" podUID="242da967-0e09-4d3c-8960-48cf07acca76" Sep 06 06:07:12.488946 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.488918 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-fksjm"] Sep 06 06:07:12.492260 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.492239 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-fksjm" Sep 06 06:07:12.494962 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.494937 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 06 06:07:12.498306 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.498282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-fksjm"] Sep 06 06:07:12.536741 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.536707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5zzq\" (UniqueName: \"kubernetes.io/projected/242da967-0e09-4d3c-8960-48cf07acca76-kube-api-access-j5zzq\") pod \"authorino-5dc7ccb5f-9nfmb\" (UID: \"242da967-0e09-4d3c-8960-48cf07acca76\") " pod="kuadrant-system/authorino-5dc7ccb5f-9nfmb" Sep 06 06:07:12.616854 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.616757 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-fksjm"] Sep 06 06:07:12.617100 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:07:12.617074 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cvc42 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-fksjm" podUID="a6b932f2-2740-483e-827d-851ca6eb08eb" Sep 06 06:07:12.637465 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.637433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6b932f2-2740-483e-827d-851ca6eb08eb-tls-cert\") pod \"authorino-687bd45585-fksjm\" (UID: \"a6b932f2-2740-483e-827d-851ca6eb08eb\") " pod="kuadrant-system/authorino-687bd45585-fksjm" Sep 06 06:07:12.637604 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.637484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5zzq\" (UniqueName: \"kubernetes.io/projected/242da967-0e09-4d3c-8960-48cf07acca76-kube-api-access-j5zzq\") pod \"authorino-5dc7ccb5f-9nfmb\" (UID: \"242da967-0e09-4d3c-8960-48cf07acca76\") " pod="kuadrant-system/authorino-5dc7ccb5f-9nfmb" Sep 06 06:07:12.637604 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.637591 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvc42\" (UniqueName: \"kubernetes.io/projected/a6b932f2-2740-483e-827d-851ca6eb08eb-kube-api-access-cvc42\") pod \"authorino-687bd45585-fksjm\" (UID: \"a6b932f2-2740-483e-827d-851ca6eb08eb\") " pod="kuadrant-system/authorino-687bd45585-fksjm" Sep 06 06:07:12.643504 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.643473 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59b8b68c49-bxpqv"] Sep 06 06:07:12.646421 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.646396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5zzq\" (UniqueName: \"kubernetes.io/projected/242da967-0e09-4d3c-8960-48cf07acca76-kube-api-access-j5zzq\") pod \"authorino-5dc7ccb5f-9nfmb\" (UID: \"242da967-0e09-4d3c-8960-48cf07acca76\") " pod="kuadrant-system/authorino-5dc7ccb5f-9nfmb" Sep 06 06:07:12.646976 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.646958 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59b8b68c49-bxpqv" Sep 06 06:07:12.655257 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.655234 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59b8b68c49-bxpqv"] Sep 06 06:07:12.739051 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.739017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6b932f2-2740-483e-827d-851ca6eb08eb-tls-cert\") pod \"authorino-687bd45585-fksjm\" (UID: \"a6b932f2-2740-483e-827d-851ca6eb08eb\") " pod="kuadrant-system/authorino-687bd45585-fksjm" Sep 06 06:07:12.739209 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.739056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8f98\" (UniqueName: \"kubernetes.io/projected/762fd3b2-0f32-439b-a364-5ac57486154d-kube-api-access-g8f98\") pod \"authorino-59b8b68c49-bxpqv\" (UID: \"762fd3b2-0f32-439b-a364-5ac57486154d\") " pod="kuadrant-system/authorino-59b8b68c49-bxpqv" Sep 06 06:07:12.739209 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.739116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvc42\" (UniqueName: \"kubernetes.io/projected/a6b932f2-2740-483e-827d-851ca6eb08eb-kube-api-access-cvc42\") pod \"authorino-687bd45585-fksjm\" (UID: \"a6b932f2-2740-483e-827d-851ca6eb08eb\") " pod="kuadrant-system/authorino-687bd45585-fksjm" Sep 06 06:07:12.739209 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.739140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/762fd3b2-0f32-439b-a364-5ac57486154d-tls-cert\") pod \"authorino-59b8b68c49-bxpqv\" (UID: \"762fd3b2-0f32-439b-a364-5ac57486154d\") " pod="kuadrant-system/authorino-59b8b68c49-bxpqv" Sep 06 06:07:12.741737 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.741708 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6b932f2-2740-483e-827d-851ca6eb08eb-tls-cert\") pod \"authorino-687bd45585-fksjm\" (UID: \"a6b932f2-2740-483e-827d-851ca6eb08eb\") " pod="kuadrant-system/authorino-687bd45585-fksjm" Sep 06 06:07:12.746674 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.746652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvc42\" (UniqueName: \"kubernetes.io/projected/a6b932f2-2740-483e-827d-851ca6eb08eb-kube-api-access-cvc42\") pod \"authorino-687bd45585-fksjm\" (UID: \"a6b932f2-2740-483e-827d-851ca6eb08eb\") " pod="kuadrant-system/authorino-687bd45585-fksjm" Sep 06 06:07:12.840590 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.840557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/762fd3b2-0f32-439b-a364-5ac57486154d-tls-cert\") pod \"authorino-59b8b68c49-bxpqv\" (UID: \"762fd3b2-0f32-439b-a364-5ac57486154d\") " pod="kuadrant-system/authorino-59b8b68c49-bxpqv" Sep 06 06:07:12.840759 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.840609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8f98\" (UniqueName: \"kubernetes.io/projected/762fd3b2-0f32-439b-a364-5ac57486154d-kube-api-access-g8f98\") pod \"authorino-59b8b68c49-bxpqv\" (UID: \"762fd3b2-0f32-439b-a364-5ac57486154d\") " pod="kuadrant-system/authorino-59b8b68c49-bxpqv" Sep 06 06:07:12.843103 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.843082 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/762fd3b2-0f32-439b-a364-5ac57486154d-tls-cert\") pod \"authorino-59b8b68c49-bxpqv\" (UID: \"762fd3b2-0f32-439b-a364-5ac57486154d\") " pod="kuadrant-system/authorino-59b8b68c49-bxpqv" Sep 06 06:07:12.848227 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.848204 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8f98\" (UniqueName: \"kubernetes.io/projected/762fd3b2-0f32-439b-a364-5ac57486154d-kube-api-access-g8f98\") pod \"authorino-59b8b68c49-bxpqv\" (UID: \"762fd3b2-0f32-439b-a364-5ac57486154d\") " pod="kuadrant-system/authorino-59b8b68c49-bxpqv" Sep 06 06:07:12.959632 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.959602 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-fksjm" Sep 06 06:07:12.959797 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.959602 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-9nfmb" Sep 06 06:07:12.963443 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.963426 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59b8b68c49-bxpqv" Sep 06 06:07:12.967774 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.967755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-fksjm" Sep 06 06:07:12.970141 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:12.970115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-9nfmb" Sep 06 06:07:13.143379 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.143342 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5zzq\" (UniqueName: \"kubernetes.io/projected/242da967-0e09-4d3c-8960-48cf07acca76-kube-api-access-j5zzq\") pod \"242da967-0e09-4d3c-8960-48cf07acca76\" (UID: \"242da967-0e09-4d3c-8960-48cf07acca76\") " Sep 06 06:07:13.143540 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.143396 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6b932f2-2740-483e-827d-851ca6eb08eb-tls-cert\") pod \"a6b932f2-2740-483e-827d-851ca6eb08eb\" (UID: \"a6b932f2-2740-483e-827d-851ca6eb08eb\") " Sep 06 06:07:13.143540 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.143490 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvc42\" (UniqueName: \"kubernetes.io/projected/a6b932f2-2740-483e-827d-851ca6eb08eb-kube-api-access-cvc42\") pod \"a6b932f2-2740-483e-827d-851ca6eb08eb\" (UID: \"a6b932f2-2740-483e-827d-851ca6eb08eb\") " Sep 06 06:07:13.145424 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.145393 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242da967-0e09-4d3c-8960-48cf07acca76-kube-api-access-j5zzq" (OuterVolumeSpecName: "kube-api-access-j5zzq") pod "242da967-0e09-4d3c-8960-48cf07acca76" (UID: "242da967-0e09-4d3c-8960-48cf07acca76"). InnerVolumeSpecName "kube-api-access-j5zzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:07:13.145536 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.145489 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b932f2-2740-483e-827d-851ca6eb08eb-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a6b932f2-2740-483e-827d-851ca6eb08eb" (UID: "a6b932f2-2740-483e-827d-851ca6eb08eb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:07:13.145536 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.145507 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b932f2-2740-483e-827d-851ca6eb08eb-kube-api-access-cvc42" (OuterVolumeSpecName: "kube-api-access-cvc42") pod "a6b932f2-2740-483e-827d-851ca6eb08eb" (UID: "a6b932f2-2740-483e-827d-851ca6eb08eb"). InnerVolumeSpecName "kube-api-access-cvc42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:07:13.230386 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.230355 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59b8b68c49-bxpqv"] Sep 06 06:07:13.232610 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:07:13.232587 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod762fd3b2_0f32_439b_a364_5ac57486154d.slice/crio-7aa2d122490d2e8401d90bb596fe517fde6e1321cd1bd57906fdf587c2ef2d6d WatchSource:0}: Error finding container 7aa2d122490d2e8401d90bb596fe517fde6e1321cd1bd57906fdf587c2ef2d6d: Status 404 returned error can't find the container with id 7aa2d122490d2e8401d90bb596fe517fde6e1321cd1bd57906fdf587c2ef2d6d Sep 06 06:07:13.244836 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.244806 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvc42\" (UniqueName: \"kubernetes.io/projected/a6b932f2-2740-483e-827d-851ca6eb08eb-kube-api-access-cvc42\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:07:13.244922 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.244842 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5zzq\" (UniqueName: \"kubernetes.io/projected/242da967-0e09-4d3c-8960-48cf07acca76-kube-api-access-j5zzq\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:07:13.244922 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.244857 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6b932f2-2740-483e-827d-851ca6eb08eb-tls-cert\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:07:13.966085 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.966052 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59b8b68c49-bxpqv" event={"ID":"762fd3b2-0f32-439b-a364-5ac57486154d","Type":"ContainerStarted","Data":"7aa2d122490d2e8401d90bb596fe517fde6e1321cd1bd57906fdf587c2ef2d6d"} Sep 06 06:07:13.966447 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.966100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-fksjm" Sep 06 06:07:13.966447 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:13.966101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-9nfmb" Sep 06 06:07:14.002265 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:14.002233 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9nfmb"] Sep 06 06:07:14.005891 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:14.005865 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9nfmb"] Sep 06 06:07:14.026313 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:14.026277 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-fksjm"] Sep 06 06:07:14.029425 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:14.029399 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-fksjm"] Sep 06 06:07:14.875146 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:14.875100 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="242da967-0e09-4d3c-8960-48cf07acca76" path="/var/lib/kubelet/pods/242da967-0e09-4d3c-8960-48cf07acca76/volumes" Sep 06 06:07:14.875477 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:14.875457 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6b932f2-2740-483e-827d-851ca6eb08eb" path="/var/lib/kubelet/pods/a6b932f2-2740-483e-827d-851ca6eb08eb/volumes" Sep 06 06:07:14.972040 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:14.971998 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59b8b68c49-bxpqv" event={"ID":"762fd3b2-0f32-439b-a364-5ac57486154d","Type":"ContainerStarted","Data":"18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6"} Sep 06 06:07:14.987482 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:14.987433 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59b8b68c49-bxpqv" podStartSLOduration=2.31644359 podStartE2EDuration="2.98741722s" totalImagesPullingTime="670.97363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:07:14.987040292 +0000 UTC m=+800.635659470" watchObservedRunningTime="2026-09-06 06:07:14.98741722 +0000 UTC m=+800.636036398" Sep 06 06:07:15.013593 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.013559 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-rcwt7"] Sep 06 06:07:15.013817 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.013789 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-rcwt7" podUID="91929dda-9d6d-4225-bb28-b75f42fe6dcb" containerName="authorino" containerID="cri-o://7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb" gracePeriod=30 Sep 06 06:07:15.260149 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.260124 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-rcwt7" Sep 06 06:07:15.363616 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.363580 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j52hb\" (UniqueName: \"kubernetes.io/projected/91929dda-9d6d-4225-bb28-b75f42fe6dcb-kube-api-access-j52hb\") pod \"91929dda-9d6d-4225-bb28-b75f42fe6dcb\" (UID: \"91929dda-9d6d-4225-bb28-b75f42fe6dcb\") " Sep 06 06:07:15.365674 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.365645 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91929dda-9d6d-4225-bb28-b75f42fe6dcb-kube-api-access-j52hb" (OuterVolumeSpecName: "kube-api-access-j52hb") pod "91929dda-9d6d-4225-bb28-b75f42fe6dcb" (UID: "91929dda-9d6d-4225-bb28-b75f42fe6dcb"). InnerVolumeSpecName "kube-api-access-j52hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:07:15.465032 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.465000 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j52hb\" (UniqueName: \"kubernetes.io/projected/91929dda-9d6d-4225-bb28-b75f42fe6dcb-kube-api-access-j52hb\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:07:15.978246 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.978212 2580 generic.go:356] "Generic (PLEG): container finished" podID="91929dda-9d6d-4225-bb28-b75f42fe6dcb" containerID="7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb" exitCode=0 Sep 06 06:07:15.978707 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.978253 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-rcwt7" Sep 06 06:07:15.978707 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.978294 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-rcwt7" event={"ID":"91929dda-9d6d-4225-bb28-b75f42fe6dcb","Type":"ContainerDied","Data":"7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb"} Sep 06 06:07:15.978707 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.978335 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-rcwt7" event={"ID":"91929dda-9d6d-4225-bb28-b75f42fe6dcb","Type":"ContainerDied","Data":"86f4c9966e855a655f242fb8e7d6a06f7521d5283557091064502806a25c2cc5"} Sep 06 06:07:15.978707 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.978352 2580 scope.go:117] "RemoveContainer" containerID="7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb" Sep 06 06:07:15.986942 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.986919 2580 scope.go:117] "RemoveContainer" containerID="7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb" Sep 06 06:07:15.987204 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:07:15.987186 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb\": container with ID starting with 7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb not found: ID does not exist" containerID="7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb" Sep 06 06:07:15.987254 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.987212 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb"} err="failed to get container status \"7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb\": rpc error: code = NotFound desc = could not find container \"7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb\": container with ID starting with 7a8376a9738488efc9840b9b68d7f5cd1d323d24ba11ec23d20e177cf6bcebcb not found: ID does not exist" Sep 06 06:07:15.998998 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:15.998961 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-rcwt7"] Sep 06 06:07:16.002497 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:16.002478 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-rcwt7"] Sep 06 06:07:16.874321 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:16.874292 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91929dda-9d6d-4225-bb28-b75f42fe6dcb" path="/var/lib/kubelet/pods/91929dda-9d6d-4225-bb28-b75f42fe6dcb/volumes" Sep 06 06:07:56.665228 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.665189 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-95mv6"] Sep 06 06:07:56.665614 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.665542 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91929dda-9d6d-4225-bb28-b75f42fe6dcb" containerName="authorino" Sep 06 06:07:56.665614 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.665553 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="91929dda-9d6d-4225-bb28-b75f42fe6dcb" containerName="authorino" Sep 06 06:07:56.665684 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.665622 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="91929dda-9d6d-4225-bb28-b75f42fe6dcb" containerName="authorino" Sep 06 06:07:56.668820 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.668803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" Sep 06 06:07:56.672435 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.672414 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-r7msj\"" Sep 06 06:07:56.672553 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.672434 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 06 06:07:56.679027 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.679004 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-95mv6"] Sep 06 06:07:56.818197 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.818161 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzc7\" (UniqueName: \"kubernetes.io/projected/287f1091-aa00-4497-badd-63c872f52fd6-kube-api-access-cnzc7\") pod \"payload-processing-5b54b7fbb7-95mv6\" (UID: \"287f1091-aa00-4497-badd-63c872f52fd6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" Sep 06 06:07:56.818197 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.818202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287f1091-aa00-4497-badd-63c872f52fd6-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-95mv6\" (UID: \"287f1091-aa00-4497-badd-63c872f52fd6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" Sep 06 06:07:56.919126 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.919037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzc7\" (UniqueName: \"kubernetes.io/projected/287f1091-aa00-4497-badd-63c872f52fd6-kube-api-access-cnzc7\") pod \"payload-processing-5b54b7fbb7-95mv6\" (UID: \"287f1091-aa00-4497-badd-63c872f52fd6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" Sep 06 06:07:56.919126 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.919075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287f1091-aa00-4497-badd-63c872f52fd6-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-95mv6\" (UID: \"287f1091-aa00-4497-badd-63c872f52fd6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" Sep 06 06:07:56.919786 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.919761 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287f1091-aa00-4497-badd-63c872f52fd6-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-95mv6\" (UID: \"287f1091-aa00-4497-badd-63c872f52fd6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" Sep 06 06:07:56.927353 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.927323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzc7\" (UniqueName: \"kubernetes.io/projected/287f1091-aa00-4497-badd-63c872f52fd6-kube-api-access-cnzc7\") pod \"payload-processing-5b54b7fbb7-95mv6\" (UID: \"287f1091-aa00-4497-badd-63c872f52fd6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" Sep 06 06:07:56.981323 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:56.981284 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" Sep 06 06:07:57.253926 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:57.253897 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-95mv6"] Sep 06 06:07:57.256214 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:07:57.256186 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287f1091_aa00_4497_badd_63c872f52fd6.slice/crio-b44c0ee9de8ce38995ff8918e3abf7deefec315db9bcf27abf15f24501705221 WatchSource:0}: Error finding container b44c0ee9de8ce38995ff8918e3abf7deefec315db9bcf27abf15f24501705221: Status 404 returned error can't find the container with id b44c0ee9de8ce38995ff8918e3abf7deefec315db9bcf27abf15f24501705221 Sep 06 06:07:58.189345 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:58.189304 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" event={"ID":"287f1091-aa00-4497-badd-63c872f52fd6","Type":"ContainerStarted","Data":"b44c0ee9de8ce38995ff8918e3abf7deefec315db9bcf27abf15f24501705221"} Sep 06 06:07:59.196309 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:59.196274 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" event={"ID":"287f1091-aa00-4497-badd-63c872f52fd6","Type":"ContainerStarted","Data":"cade567fc00006d42da4c1442ff4b782f9b75c3590cc611cef195516f22163e6"} Sep 06 06:07:59.196753 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:59.196352 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" Sep 06 06:07:59.213740 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:07:59.213687 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" podStartSLOduration=1.68053602 podStartE2EDuration="3.213671612s" totalImagesPullingTime="1.533135592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:07:59.212637386 +0000 UTC m=+844.861256566" watchObservedRunningTime="2026-09-06 06:07:59.213671612 +0000 UTC m=+844.862290792" Sep 06 06:08:10.203019 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:10.202989 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-95mv6" Sep 06 06:08:29.896300 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:29.896266 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x"] Sep 06 06:08:29.899884 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:29.899818 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:29.902659 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:29.902638 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 06 06:08:29.903838 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:29.903809 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 06 06:08:29.903942 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:29.903843 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nbtvj\"" Sep 06 06:08:29.904120 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:29.904103 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 06 06:08:29.909886 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:29.909805 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x"] Sep 06 06:08:30.007393 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.007359 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/466a57ef-825b-41d2-9355-e1cc6cea3d03-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.007552 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.007406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/466a57ef-825b-41d2-9355-e1cc6cea3d03-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.007552 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.007456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkdh\" (UniqueName: \"kubernetes.io/projected/466a57ef-825b-41d2-9355-e1cc6cea3d03-kube-api-access-ggkdh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.007552 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.007486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/466a57ef-825b-41d2-9355-e1cc6cea3d03-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.007552 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.007510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/466a57ef-825b-41d2-9355-e1cc6cea3d03-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.007733 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.007555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/466a57ef-825b-41d2-9355-e1cc6cea3d03-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.108253 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.108217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkdh\" (UniqueName: \"kubernetes.io/projected/466a57ef-825b-41d2-9355-e1cc6cea3d03-kube-api-access-ggkdh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.108253 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.108262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/466a57ef-825b-41d2-9355-e1cc6cea3d03-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.108498 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.108284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/466a57ef-825b-41d2-9355-e1cc6cea3d03-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.108498 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.108326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/466a57ef-825b-41d2-9355-e1cc6cea3d03-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.108498 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.108371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/466a57ef-825b-41d2-9355-e1cc6cea3d03-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.108498 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.108394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/466a57ef-825b-41d2-9355-e1cc6cea3d03-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.109044 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.109012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/466a57ef-825b-41d2-9355-e1cc6cea3d03-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.109174 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.109101 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/466a57ef-825b-41d2-9355-e1cc6cea3d03-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.109389 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.109369 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/466a57ef-825b-41d2-9355-e1cc6cea3d03-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.111714 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.111687 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/466a57ef-825b-41d2-9355-e1cc6cea3d03-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.111980 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.111957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/466a57ef-825b-41d2-9355-e1cc6cea3d03-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.117651 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.117597 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkdh\" (UniqueName: \"kubernetes.io/projected/466a57ef-825b-41d2-9355-e1cc6cea3d03-kube-api-access-ggkdh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x\" (UID: \"466a57ef-825b-41d2-9355-e1cc6cea3d03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.211915 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.211881 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:30.479505 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:30.479483 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x"] Sep 06 06:08:30.481863 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:08:30.481807 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466a57ef_825b_41d2_9355_e1cc6cea3d03.slice/crio-86baff8c821e8ecc28571c91a0056addd7508810d080afae9508f178d5f5b721 WatchSource:0}: Error finding container 86baff8c821e8ecc28571c91a0056addd7508810d080afae9508f178d5f5b721: Status 404 returned error can't find the container with id 86baff8c821e8ecc28571c91a0056addd7508810d080afae9508f178d5f5b721 Sep 06 06:08:31.350501 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:31.350457 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" event={"ID":"466a57ef-825b-41d2-9355-e1cc6cea3d03","Type":"ContainerStarted","Data":"86baff8c821e8ecc28571c91a0056addd7508810d080afae9508f178d5f5b721"} Sep 06 06:08:32.356348 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:32.356308 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" event={"ID":"466a57ef-825b-41d2-9355-e1cc6cea3d03","Type":"ContainerStarted","Data":"ff8de21544852d9f62e6d1b056a9f50b0c7b7018e18589d1e4564175e35043bc"} Sep 06 06:08:32.373296 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:32.373242 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" podStartSLOduration=2.3131618 podStartE2EDuration="3.373221893s" totalImagesPullingTime="1.060060093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:08:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:08:32.372651197 +0000 UTC m=+878.021270376" watchObservedRunningTime="2026-09-06 06:08:32.373221893 +0000 UTC m=+878.021841073" Sep 06 06:08:40.212686 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:40.212646 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:40.213095 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:40.212724 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:40.224972 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:40.224940 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:40.405532 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:40.405497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x" Sep 06 06:08:47.014220 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.014184 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-thlff"] Sep 06 06:08:47.018230 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.018204 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.020982 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.020962 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 06 06:08:47.031344 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.031321 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-thlff"] Sep 06 06:08:47.170159 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.170117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9966d16e-844e-4b92-928b-e110d04c0426-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.170159 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.170165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9966d16e-844e-4b92-928b-e110d04c0426-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.170392 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.170187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9966d16e-844e-4b92-928b-e110d04c0426-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.170392 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.170262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9966d16e-844e-4b92-928b-e110d04c0426-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.170392 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.170301 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mckzz\" (UniqueName: \"kubernetes.io/projected/9966d16e-844e-4b92-928b-e110d04c0426-kube-api-access-mckzz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.170392 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.170354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9966d16e-844e-4b92-928b-e110d04c0426-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.271747 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.271631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9966d16e-844e-4b92-928b-e110d04c0426-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.271747 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.271678 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mckzz\" (UniqueName: \"kubernetes.io/projected/9966d16e-844e-4b92-928b-e110d04c0426-kube-api-access-mckzz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.271747 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.271734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9966d16e-844e-4b92-928b-e110d04c0426-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.272125 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.271763 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9966d16e-844e-4b92-928b-e110d04c0426-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.272125 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.271784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9966d16e-844e-4b92-928b-e110d04c0426-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.272125 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.271815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9966d16e-844e-4b92-928b-e110d04c0426-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.272287 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.272201 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9966d16e-844e-4b92-928b-e110d04c0426-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.272287 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.272239 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9966d16e-844e-4b92-928b-e110d04c0426-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.272287 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.272246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9966d16e-844e-4b92-928b-e110d04c0426-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.273979 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.273959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9966d16e-844e-4b92-928b-e110d04c0426-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.274288 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.274272 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9966d16e-844e-4b92-928b-e110d04c0426-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.283176 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.283155 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mckzz\" (UniqueName: \"kubernetes.io/projected/9966d16e-844e-4b92-928b-e110d04c0426-kube-api-access-mckzz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-thlff\" (UID: \"9966d16e-844e-4b92-928b-e110d04c0426\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.355342 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.355309 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:47.627680 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:47.627650 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-thlff"] Sep 06 06:08:47.629559 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:08:47.629531 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9966d16e_844e_4b92_928b_e110d04c0426.slice/crio-8e57e2c5cb8f8ab41ef98cd8c0826e2aeeb4e68c5cf67b68e3c72c1c53a31d34 WatchSource:0}: Error finding container 8e57e2c5cb8f8ab41ef98cd8c0826e2aeeb4e68c5cf67b68e3c72c1c53a31d34: Status 404 returned error can't find the container with id 8e57e2c5cb8f8ab41ef98cd8c0826e2aeeb4e68c5cf67b68e3c72c1c53a31d34 Sep 06 06:08:48.435304 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:48.435265 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" event={"ID":"9966d16e-844e-4b92-928b-e110d04c0426","Type":"ContainerStarted","Data":"05685fbad234f88b337d0f88f7937805a371c04daf6ac382fed2aa2be0a6ac0d"} Sep 06 06:08:48.435715 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:48.435319 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" event={"ID":"9966d16e-844e-4b92-928b-e110d04c0426","Type":"ContainerStarted","Data":"8e57e2c5cb8f8ab41ef98cd8c0826e2aeeb4e68c5cf67b68e3c72c1c53a31d34"} Sep 06 06:08:48.453630 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:48.453577 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" podStartSLOduration=2.2922741699999998 podStartE2EDuration="2.453560923s" totalImagesPullingTime="161.286753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:08:48.451033967 +0000 UTC m=+894.099653149" watchObservedRunningTime="2026-09-06 06:08:48.453560923 +0000 UTC m=+894.102180102" Sep 06 06:08:57.355691 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:57.355560 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:57.355691 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:57.355622 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:57.376224 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:57.368193 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:08:57.490181 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:08:57.490145 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-thlff" Sep 06 06:09:04.410185 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.410136 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c"] Sep 06 06:09:04.413973 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.413951 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.416543 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.416523 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 06 06:09:04.423183 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.423160 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c"] Sep 06 06:09:04.423794 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.423768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22be0ec4-6100-49d7-a2c2-5caac18d8d14-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.423914 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.423801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be0ec4-6100-49d7-a2c2-5caac18d8d14-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.423914 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.423846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22be0ec4-6100-49d7-a2c2-5caac18d8d14-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.424029 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.423967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be0ec4-6100-49d7-a2c2-5caac18d8d14-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.424029 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.424008 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tshc\" (UniqueName: \"kubernetes.io/projected/22be0ec4-6100-49d7-a2c2-5caac18d8d14-kube-api-access-7tshc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.424124 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.424030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be0ec4-6100-49d7-a2c2-5caac18d8d14-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.525142 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.525108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be0ec4-6100-49d7-a2c2-5caac18d8d14-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.525321 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.525151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tshc\" (UniqueName: \"kubernetes.io/projected/22be0ec4-6100-49d7-a2c2-5caac18d8d14-kube-api-access-7tshc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.525321 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.525174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be0ec4-6100-49d7-a2c2-5caac18d8d14-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.525321 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.525207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22be0ec4-6100-49d7-a2c2-5caac18d8d14-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.525321 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.525226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be0ec4-6100-49d7-a2c2-5caac18d8d14-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.525321 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.525254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22be0ec4-6100-49d7-a2c2-5caac18d8d14-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.525613 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.525589 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be0ec4-6100-49d7-a2c2-5caac18d8d14-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.525678 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.525624 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be0ec4-6100-49d7-a2c2-5caac18d8d14-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.525678 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.525668 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22be0ec4-6100-49d7-a2c2-5caac18d8d14-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.527484 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.527450 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be0ec4-6100-49d7-a2c2-5caac18d8d14-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.527662 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.527647 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22be0ec4-6100-49d7-a2c2-5caac18d8d14-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.532149 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.532124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tshc\" (UniqueName: \"kubernetes.io/projected/22be0ec4-6100-49d7-a2c2-5caac18d8d14-kube-api-access-7tshc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ww76c\" (UID: \"22be0ec4-6100-49d7-a2c2-5caac18d8d14\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.724984 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.724941 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:04.995067 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.995043 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c"] Sep 06 06:09:04.997266 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:09:04.997235 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22be0ec4_6100_49d7_a2c2_5caac18d8d14.slice/crio-4f658988c5e46ac3aa5ed1b8308f533107913ab638fabd86b98e157b0d76a0ea WatchSource:0}: Error finding container 4f658988c5e46ac3aa5ed1b8308f533107913ab638fabd86b98e157b0d76a0ea: Status 404 returned error can't find the container with id 4f658988c5e46ac3aa5ed1b8308f533107913ab638fabd86b98e157b0d76a0ea Sep 06 06:09:04.998712 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:04.998695 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 06:09:05.519373 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:05.519331 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" event={"ID":"22be0ec4-6100-49d7-a2c2-5caac18d8d14","Type":"ContainerStarted","Data":"b0bf796146da6e81ff6bd7b164a9c09be29c5fbe9d0921954f640b0987f20f5c"} Sep 06 06:09:05.519776 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:05.519409 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" event={"ID":"22be0ec4-6100-49d7-a2c2-5caac18d8d14","Type":"ContainerStarted","Data":"4f658988c5e46ac3aa5ed1b8308f533107913ab638fabd86b98e157b0d76a0ea"} Sep 06 06:09:05.537213 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:05.537166 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" podStartSLOduration=1.367692801 podStartE2EDuration="1.537150024s" totalImagesPullingTime="169.457223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:09:05.535392614 +0000 UTC m=+911.184011794" watchObservedRunningTime="2026-09-06 06:09:05.537150024 +0000 UTC m=+911.185769202" Sep 06 06:09:14.725296 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:14.725260 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:14.725790 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:14.725607 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:14.737877 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:14.737852 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:15.579258 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:15.579226 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ww76c" Sep 06 06:09:33.505444 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.505406 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b"] Sep 06 06:09:33.509539 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.509514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.512318 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.512289 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 06 06:09:33.518812 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.518788 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b"] Sep 06 06:09:33.592337 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.592236 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fa6afd-4651-429c-adb9-1858950002bc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.592337 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.592299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fa6afd-4651-429c-adb9-1858950002bc-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.592559 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.592357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fa6afd-4651-429c-adb9-1858950002bc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.592559 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.592397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fa6afd-4651-429c-adb9-1858950002bc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.592559 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.592505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fa6afd-4651-429c-adb9-1858950002bc-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.592559 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.592548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbrxj\" (UniqueName: \"kubernetes.io/projected/54fa6afd-4651-429c-adb9-1858950002bc-kube-api-access-bbrxj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.693479 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.693438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fa6afd-4651-429c-adb9-1858950002bc-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.693479 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.693481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fa6afd-4651-429c-adb9-1858950002bc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.693734 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.693508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fa6afd-4651-429c-adb9-1858950002bc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.693734 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.693567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fa6afd-4651-429c-adb9-1858950002bc-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.693734 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.693584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrxj\" (UniqueName: \"kubernetes.io/projected/54fa6afd-4651-429c-adb9-1858950002bc-kube-api-access-bbrxj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.693936 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.693746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fa6afd-4651-429c-adb9-1858950002bc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.693995 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.693957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fa6afd-4651-429c-adb9-1858950002bc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.694038 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.694012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fa6afd-4651-429c-adb9-1858950002bc-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.694073 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.694036 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fa6afd-4651-429c-adb9-1858950002bc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.695785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.695764 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fa6afd-4651-429c-adb9-1858950002bc-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.696131 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.696113 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fa6afd-4651-429c-adb9-1858950002bc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.701253 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.701230 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrxj\" (UniqueName: \"kubernetes.io/projected/54fa6afd-4651-429c-adb9-1858950002bc-kube-api-access-bbrxj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b\" (UID: \"54fa6afd-4651-429c-adb9-1858950002bc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:33.820364 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:33.820318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:34.088669 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:34.088581 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b"] Sep 06 06:09:34.091110 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:09:34.091077 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54fa6afd_4651_429c_adb9_1858950002bc.slice/crio-6a4a0b77dbc39715111ec04e136fb06199b03884ce8c255c047031ae3d7eb406 WatchSource:0}: Error finding container 6a4a0b77dbc39715111ec04e136fb06199b03884ce8c255c047031ae3d7eb406: Status 404 returned error can't find the container with id 6a4a0b77dbc39715111ec04e136fb06199b03884ce8c255c047031ae3d7eb406 Sep 06 06:09:34.662203 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:34.662102 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" event={"ID":"54fa6afd-4651-429c-adb9-1858950002bc","Type":"ContainerStarted","Data":"1beb46715d0cad2a440f2ccc7199a85aee8b8f90aa557554819c46d35e8b56ee"} Sep 06 06:09:34.662203 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:34.662144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" event={"ID":"54fa6afd-4651-429c-adb9-1858950002bc","Type":"ContainerStarted","Data":"6a4a0b77dbc39715111ec04e136fb06199b03884ce8c255c047031ae3d7eb406"} Sep 06 06:09:34.680345 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:34.680267 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" podStartSLOduration=1.511964722 podStartE2EDuration="1.680248574s" totalImagesPullingTime="168.283852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:09:34.678879042 +0000 UTC m=+940.327498223" watchObservedRunningTime="2026-09-06 06:09:34.680248574 +0000 UTC m=+940.328867753" Sep 06 06:09:43.820653 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:43.820616 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:43.821174 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:43.820666 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:43.832821 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:43.832789 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:09:44.724772 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:09:44.724734 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b" Sep 06 06:10:31.051623 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:31.051543 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d6d57f484-zh75v"] Sep 06 06:10:31.055073 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:31.055056 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d6d57f484-zh75v" Sep 06 06:10:31.062523 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:31.062335 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d6d57f484-zh75v"] Sep 06 06:10:31.202451 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:31.202419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3caa196-731c-4c20-85ea-c52c1d4a465a-tls-cert\") pod \"authorino-d6d57f484-zh75v\" (UID: \"f3caa196-731c-4c20-85ea-c52c1d4a465a\") " pod="kuadrant-system/authorino-d6d57f484-zh75v" Sep 06 06:10:31.202607 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:31.202461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s27rs\" (UniqueName: \"kubernetes.io/projected/f3caa196-731c-4c20-85ea-c52c1d4a465a-kube-api-access-s27rs\") pod \"authorino-d6d57f484-zh75v\" (UID: \"f3caa196-731c-4c20-85ea-c52c1d4a465a\") " pod="kuadrant-system/authorino-d6d57f484-zh75v" Sep 06 06:10:31.303378 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:31.303286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3caa196-731c-4c20-85ea-c52c1d4a465a-tls-cert\") pod \"authorino-d6d57f484-zh75v\" (UID: \"f3caa196-731c-4c20-85ea-c52c1d4a465a\") " pod="kuadrant-system/authorino-d6d57f484-zh75v" Sep 06 06:10:31.303378 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:31.303341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s27rs\" (UniqueName: \"kubernetes.io/projected/f3caa196-731c-4c20-85ea-c52c1d4a465a-kube-api-access-s27rs\") pod \"authorino-d6d57f484-zh75v\" (UID: \"f3caa196-731c-4c20-85ea-c52c1d4a465a\") " pod="kuadrant-system/authorino-d6d57f484-zh75v" Sep 06 06:10:31.305765 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:31.305742 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3caa196-731c-4c20-85ea-c52c1d4a465a-tls-cert\") pod \"authorino-d6d57f484-zh75v\" (UID: \"f3caa196-731c-4c20-85ea-c52c1d4a465a\") " pod="kuadrant-system/authorino-d6d57f484-zh75v" Sep 06 06:10:31.311823 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:31.311797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s27rs\" (UniqueName: \"kubernetes.io/projected/f3caa196-731c-4c20-85ea-c52c1d4a465a-kube-api-access-s27rs\") pod \"authorino-d6d57f484-zh75v\" (UID: \"f3caa196-731c-4c20-85ea-c52c1d4a465a\") " pod="kuadrant-system/authorino-d6d57f484-zh75v" Sep 06 06:10:31.364807 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:31.364776 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d6d57f484-zh75v" Sep 06 06:10:31.633785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:31.633759 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d6d57f484-zh75v"] Sep 06 06:10:31.636368 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:10:31.636340 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3caa196_731c_4c20_85ea_c52c1d4a465a.slice/crio-24fff4d42fc720059746b9217ba5ab66076e6f7621ecf1380894358d7b8fc230 WatchSource:0}: Error finding container 24fff4d42fc720059746b9217ba5ab66076e6f7621ecf1380894358d7b8fc230: Status 404 returned error can't find the container with id 24fff4d42fc720059746b9217ba5ab66076e6f7621ecf1380894358d7b8fc230 Sep 06 06:10:31.943190 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:31.943155 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d6d57f484-zh75v" event={"ID":"f3caa196-731c-4c20-85ea-c52c1d4a465a","Type":"ContainerStarted","Data":"24fff4d42fc720059746b9217ba5ab66076e6f7621ecf1380894358d7b8fc230"} Sep 06 06:10:32.952412 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:32.952365 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d6d57f484-zh75v" event={"ID":"f3caa196-731c-4c20-85ea-c52c1d4a465a","Type":"ContainerStarted","Data":"1b5a56b2aee7eb9b9d0b526375a8ecc8c919a85572ab38992b6f0d3d892ad1eb"} Sep 06 06:10:32.968024 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:32.967961 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d6d57f484-zh75v" podStartSLOduration=1.118545635 podStartE2EDuration="1.967937879s" totalImagesPullingTime="849.392244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:10:32.96669213 +0000 UTC m=+998.615311310" watchObservedRunningTime="2026-09-06 06:10:32.967937879 +0000 UTC m=+998.616557059" Sep 06 06:10:33.004295 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.004216 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59b8b68c49-bxpqv"] Sep 06 06:10:33.004492 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.004402 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59b8b68c49-bxpqv" podUID="762fd3b2-0f32-439b-a364-5ac57486154d" containerName="authorino" containerID="cri-o://18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6" gracePeriod=30 Sep 06 06:10:33.262900 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.262874 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59b8b68c49-bxpqv" Sep 06 06:10:33.424068 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.424019 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8f98\" (UniqueName: \"kubernetes.io/projected/762fd3b2-0f32-439b-a364-5ac57486154d-kube-api-access-g8f98\") pod \"762fd3b2-0f32-439b-a364-5ac57486154d\" (UID: \"762fd3b2-0f32-439b-a364-5ac57486154d\") " Sep 06 06:10:33.424068 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.424069 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/762fd3b2-0f32-439b-a364-5ac57486154d-tls-cert\") pod \"762fd3b2-0f32-439b-a364-5ac57486154d\" (UID: \"762fd3b2-0f32-439b-a364-5ac57486154d\") " Sep 06 06:10:33.426209 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.426175 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762fd3b2-0f32-439b-a364-5ac57486154d-kube-api-access-g8f98" (OuterVolumeSpecName: "kube-api-access-g8f98") pod "762fd3b2-0f32-439b-a364-5ac57486154d" (UID: "762fd3b2-0f32-439b-a364-5ac57486154d"). InnerVolumeSpecName "kube-api-access-g8f98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:10:33.434714 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.434685 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762fd3b2-0f32-439b-a364-5ac57486154d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "762fd3b2-0f32-439b-a364-5ac57486154d" (UID: "762fd3b2-0f32-439b-a364-5ac57486154d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:10:33.525288 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.525209 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/762fd3b2-0f32-439b-a364-5ac57486154d-tls-cert\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:10:33.525288 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.525248 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8f98\" (UniqueName: \"kubernetes.io/projected/762fd3b2-0f32-439b-a364-5ac57486154d-kube-api-access-g8f98\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:10:33.959086 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.959051 2580 generic.go:356] "Generic (PLEG): container finished" podID="762fd3b2-0f32-439b-a364-5ac57486154d" containerID="18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6" exitCode=0 Sep 06 06:10:33.959503 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.959125 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59b8b68c49-bxpqv" Sep 06 06:10:33.959503 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.959143 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59b8b68c49-bxpqv" event={"ID":"762fd3b2-0f32-439b-a364-5ac57486154d","Type":"ContainerDied","Data":"18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6"} Sep 06 06:10:33.959503 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.959191 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59b8b68c49-bxpqv" event={"ID":"762fd3b2-0f32-439b-a364-5ac57486154d","Type":"ContainerDied","Data":"7aa2d122490d2e8401d90bb596fe517fde6e1321cd1bd57906fdf587c2ef2d6d"} Sep 06 06:10:33.959503 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.959209 2580 scope.go:117] "RemoveContainer" containerID="18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6" Sep 06 06:10:33.970046 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.970000 2580 scope.go:117] "RemoveContainer" containerID="18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6" Sep 06 06:10:33.970379 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:10:33.970359 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6\": container with ID starting with 18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6 not found: ID does not exist" containerID="18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6" Sep 06 06:10:33.970449 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.970388 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6"} err="failed to get container status \"18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6\": rpc error: code = NotFound desc = could not find container \"18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6\": container with ID starting with 18c1d24fc457cc07685b15f65b77541e21c21caf6e5c7d18f370730f70baaec6 not found: ID does not exist" Sep 06 06:10:33.980718 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.980693 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59b8b68c49-bxpqv"] Sep 06 06:10:33.984729 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:33.984705 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59b8b68c49-bxpqv"] Sep 06 06:10:34.875699 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:10:34.875667 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="762fd3b2-0f32-439b-a364-5ac57486154d" path="/var/lib/kubelet/pods/762fd3b2-0f32-439b-a364-5ac57486154d/volumes" Sep 06 06:11:02.790197 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.790158 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t"] Sep 06 06:11:02.790961 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.790695 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762fd3b2-0f32-439b-a364-5ac57486154d" containerName="authorino" Sep 06 06:11:02.790961 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.790715 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="762fd3b2-0f32-439b-a364-5ac57486154d" containerName="authorino" Sep 06 06:11:02.790961 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.790794 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="762fd3b2-0f32-439b-a364-5ac57486154d" containerName="authorino" Sep 06 06:11:02.795093 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.795073 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" Sep 06 06:11:02.797947 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.797925 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5da01692\"" Sep 06 06:11:02.798079 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.797927 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5da01692-dockercfg-bcgk4\"" Sep 06 06:11:02.803839 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.803796 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t"] Sep 06 06:11:02.827111 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.827082 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk"] Sep 06 06:11:02.830525 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.830509 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" Sep 06 06:11:02.840270 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.840240 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk"] Sep 06 06:11:02.895549 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.895515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347-plugins-config-volume\") pod \"payload-processing-e2e-ait-5da01692-7b5594b68c-f769t\" (UID: \"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347\") " pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" Sep 06 06:11:02.895549 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.895552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phjtl\" (UniqueName: \"kubernetes.io/projected/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347-kube-api-access-phjtl\") pod \"payload-processing-e2e-ait-5da01692-7b5594b68c-f769t\" (UID: \"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347\") " pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" Sep 06 06:11:02.895744 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.895599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj6dz\" (UniqueName: \"kubernetes.io/projected/21ff4329-7804-483e-b319-b9dbf3b2123d-kube-api-access-dj6dz\") pod \"payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk\" (UID: \"21ff4329-7804-483e-b319-b9dbf3b2123d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" Sep 06 06:11:02.895744 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.895624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ff4329-7804-483e-b319-b9dbf3b2123d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk\" (UID: \"21ff4329-7804-483e-b319-b9dbf3b2123d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" Sep 06 06:11:02.996868 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.996805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj6dz\" (UniqueName: \"kubernetes.io/projected/21ff4329-7804-483e-b319-b9dbf3b2123d-kube-api-access-dj6dz\") pod \"payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk\" (UID: \"21ff4329-7804-483e-b319-b9dbf3b2123d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" Sep 06 06:11:02.997025 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.996872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ff4329-7804-483e-b319-b9dbf3b2123d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk\" (UID: \"21ff4329-7804-483e-b319-b9dbf3b2123d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" Sep 06 06:11:02.997025 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.996983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347-plugins-config-volume\") pod \"payload-processing-e2e-ait-5da01692-7b5594b68c-f769t\" (UID: \"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347\") " pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" Sep 06 06:11:02.997025 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.997003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phjtl\" (UniqueName: \"kubernetes.io/projected/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347-kube-api-access-phjtl\") pod \"payload-processing-e2e-ait-5da01692-7b5594b68c-f769t\" (UID: \"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347\") " pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" Sep 06 06:11:02.997683 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.997663 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ff4329-7804-483e-b319-b9dbf3b2123d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk\" (UID: \"21ff4329-7804-483e-b319-b9dbf3b2123d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" Sep 06 06:11:02.997728 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:02.997710 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347-plugins-config-volume\") pod \"payload-processing-e2e-ait-5da01692-7b5594b68c-f769t\" (UID: \"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347\") " pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" Sep 06 06:11:03.004668 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:03.004645 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phjtl\" (UniqueName: \"kubernetes.io/projected/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347-kube-api-access-phjtl\") pod \"payload-processing-e2e-ait-5da01692-7b5594b68c-f769t\" (UID: \"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347\") " pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" Sep 06 06:11:03.004784 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:03.004736 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj6dz\" (UniqueName: \"kubernetes.io/projected/21ff4329-7804-483e-b319-b9dbf3b2123d-kube-api-access-dj6dz\") pod \"payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk\" (UID: \"21ff4329-7804-483e-b319-b9dbf3b2123d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" Sep 06 06:11:03.106460 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:03.106385 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" Sep 06 06:11:03.141115 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:03.141089 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" Sep 06 06:11:03.378044 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:03.378013 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t"] Sep 06 06:11:03.379935 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:11:03.379906 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d2bac2_9b87_4cfc_8d13_e40ccbbb9347.slice/crio-8ac315a327ed3f8ce047710f39042ba8488ea7ac848e8174b790380808b61d89 WatchSource:0}: Error finding container 8ac315a327ed3f8ce047710f39042ba8488ea7ac848e8174b790380808b61d89: Status 404 returned error can't find the container with id 8ac315a327ed3f8ce047710f39042ba8488ea7ac848e8174b790380808b61d89 Sep 06 06:11:03.424538 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:03.424505 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk"] Sep 06 06:11:03.437089 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:11:03.437060 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ff4329_7804_483e_b319_b9dbf3b2123d.slice/crio-1254dbd50461859c3f9a643b163ecc99e4e03f87f0c0e4cdf6513b5f7bbaba34 WatchSource:0}: Error finding container 1254dbd50461859c3f9a643b163ecc99e4e03f87f0c0e4cdf6513b5f7bbaba34: Status 404 returned error can't find the container with id 1254dbd50461859c3f9a643b163ecc99e4e03f87f0c0e4cdf6513b5f7bbaba34 Sep 06 06:11:04.111143 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:04.111104 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" event={"ID":"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347","Type":"ContainerStarted","Data":"333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3"} Sep 06 06:11:04.111678 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:04.111170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" event={"ID":"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347","Type":"ContainerStarted","Data":"8ac315a327ed3f8ce047710f39042ba8488ea7ac848e8174b790380808b61d89"} Sep 06 06:11:04.111678 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:04.111203 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" Sep 06 06:11:04.112745 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:04.112722 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" event={"ID":"21ff4329-7804-483e-b319-b9dbf3b2123d","Type":"ContainerStarted","Data":"5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700"} Sep 06 06:11:04.112852 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:04.112767 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" event={"ID":"21ff4329-7804-483e-b319-b9dbf3b2123d","Type":"ContainerStarted","Data":"1254dbd50461859c3f9a643b163ecc99e4e03f87f0c0e4cdf6513b5f7bbaba34"} Sep 06 06:11:04.112852 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:04.112803 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" Sep 06 06:11:04.129603 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:04.129557 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" podStartSLOduration=2.129541704 podStartE2EDuration="2.129541704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:11:04.127803654 +0000 UTC m=+1029.776422850" watchObservedRunningTime="2026-09-06 06:11:04.129541704 +0000 UTC m=+1029.778160881" Sep 06 06:11:04.144773 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:04.144725 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" podStartSLOduration=2.144708788 podStartE2EDuration="2.144708788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:11:04.142377882 +0000 UTC m=+1029.790997087" watchObservedRunningTime="2026-09-06 06:11:04.144708788 +0000 UTC m=+1029.793327967" Sep 06 06:11:05.407136 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.407099 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg"] Sep 06 06:11:05.411162 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.411135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" Sep 06 06:11:05.414063 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.414037 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-e75768-dockercfg-htm2j\"" Sep 06 06:11:05.414259 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.414236 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-e75768\"" Sep 06 06:11:05.421179 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.421154 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg"] Sep 06 06:11:05.521110 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.521073 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th4nh\" (UniqueName: \"kubernetes.io/projected/de73a3c3-016c-4124-82cc-c035ef46c60f-kube-api-access-th4nh\") pod \"payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg\" (UID: \"de73a3c3-016c-4124-82cc-c035ef46c60f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" Sep 06 06:11:05.521110 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.521113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de73a3c3-016c-4124-82cc-c035ef46c60f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg\" (UID: \"de73a3c3-016c-4124-82cc-c035ef46c60f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" Sep 06 06:11:05.621962 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.621927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4nh\" (UniqueName: \"kubernetes.io/projected/de73a3c3-016c-4124-82cc-c035ef46c60f-kube-api-access-th4nh\") pod \"payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg\" (UID: \"de73a3c3-016c-4124-82cc-c035ef46c60f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" Sep 06 06:11:05.621962 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.621964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de73a3c3-016c-4124-82cc-c035ef46c60f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg\" (UID: \"de73a3c3-016c-4124-82cc-c035ef46c60f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" Sep 06 06:11:05.622646 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.622625 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de73a3c3-016c-4124-82cc-c035ef46c60f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg\" (UID: \"de73a3c3-016c-4124-82cc-c035ef46c60f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" Sep 06 06:11:05.633595 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.633564 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th4nh\" (UniqueName: \"kubernetes.io/projected/de73a3c3-016c-4124-82cc-c035ef46c60f-kube-api-access-th4nh\") pod \"payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg\" (UID: \"de73a3c3-016c-4124-82cc-c035ef46c60f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" Sep 06 06:11:05.722749 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.722709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" Sep 06 06:11:05.991750 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:05.991725 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg"] Sep 06 06:11:05.993944 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:11:05.993913 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde73a3c3_016c_4124_82cc_c035ef46c60f.slice/crio-2bd5139c9fd15b586de84178552cd636c9ba4b89b50c45f3392c65f238b4ae74 WatchSource:0}: Error finding container 2bd5139c9fd15b586de84178552cd636c9ba4b89b50c45f3392c65f238b4ae74: Status 404 returned error can't find the container with id 2bd5139c9fd15b586de84178552cd636c9ba4b89b50c45f3392c65f238b4ae74 Sep 06 06:11:06.126333 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:06.126287 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" event={"ID":"de73a3c3-016c-4124-82cc-c035ef46c60f","Type":"ContainerStarted","Data":"04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a"} Sep 06 06:11:06.126492 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:06.126349 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" event={"ID":"de73a3c3-016c-4124-82cc-c035ef46c60f","Type":"ContainerStarted","Data":"2bd5139c9fd15b586de84178552cd636c9ba4b89b50c45f3392c65f238b4ae74"} Sep 06 06:11:06.126492 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:06.126374 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" Sep 06 06:11:06.141699 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:06.141649 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" podStartSLOduration=1.141635217 podStartE2EDuration="1.141635217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:11:06.140448884 +0000 UTC m=+1031.789068064" watchObservedRunningTime="2026-09-06 06:11:06.141635217 +0000 UTC m=+1031.790254439" Sep 06 06:11:15.121729 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:15.121695 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" Sep 06 06:11:15.122217 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:15.121986 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" Sep 06 06:11:17.134380 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:17.134347 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" Sep 06 06:11:23.490752 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.490615 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv"] Sep 06 06:11:23.494164 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.494143 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" Sep 06 06:11:23.497170 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.497147 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0bce0b-dockercfg-dh2gh\"" Sep 06 06:11:23.497301 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.497213 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0bce0b\"" Sep 06 06:11:23.507388 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.507361 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv"] Sep 06 06:11:23.538600 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.538570 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr"] Sep 06 06:11:23.542126 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.542106 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" Sep 06 06:11:23.549979 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.549960 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr"] Sep 06 06:11:23.594030 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.594005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97gms\" (UniqueName: \"kubernetes.io/projected/b69a40a3-85f5-4c78-b9ca-8f8980025811-kube-api-access-97gms\") pod \"payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr\" (UID: \"b69a40a3-85f5-4c78-b9ca-8f8980025811\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" Sep 06 06:11:23.594238 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.594060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6tnz\" (UniqueName: \"kubernetes.io/projected/417b0b5f-25d5-4007-bdad-91493886b9d9-kube-api-access-s6tnz\") pod \"payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv\" (UID: \"417b0b5f-25d5-4007-bdad-91493886b9d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" Sep 06 06:11:23.594238 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.594097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417b0b5f-25d5-4007-bdad-91493886b9d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv\" (UID: \"417b0b5f-25d5-4007-bdad-91493886b9d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" Sep 06 06:11:23.594238 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.594133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69a40a3-85f5-4c78-b9ca-8f8980025811-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr\" (UID: \"b69a40a3-85f5-4c78-b9ca-8f8980025811\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" Sep 06 06:11:23.694763 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.694734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97gms\" (UniqueName: \"kubernetes.io/projected/b69a40a3-85f5-4c78-b9ca-8f8980025811-kube-api-access-97gms\") pod \"payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr\" (UID: \"b69a40a3-85f5-4c78-b9ca-8f8980025811\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" Sep 06 06:11:23.694918 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.694778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6tnz\" (UniqueName: \"kubernetes.io/projected/417b0b5f-25d5-4007-bdad-91493886b9d9-kube-api-access-s6tnz\") pod \"payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv\" (UID: \"417b0b5f-25d5-4007-bdad-91493886b9d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" Sep 06 06:11:23.694918 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.694801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417b0b5f-25d5-4007-bdad-91493886b9d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv\" (UID: \"417b0b5f-25d5-4007-bdad-91493886b9d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" Sep 06 06:11:23.694918 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.694865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69a40a3-85f5-4c78-b9ca-8f8980025811-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr\" (UID: \"b69a40a3-85f5-4c78-b9ca-8f8980025811\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" Sep 06 06:11:23.695552 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.695533 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69a40a3-85f5-4c78-b9ca-8f8980025811-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr\" (UID: \"b69a40a3-85f5-4c78-b9ca-8f8980025811\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" Sep 06 06:11:23.695594 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.695546 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417b0b5f-25d5-4007-bdad-91493886b9d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv\" (UID: \"417b0b5f-25d5-4007-bdad-91493886b9d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" Sep 06 06:11:23.702860 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.702816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6tnz\" (UniqueName: \"kubernetes.io/projected/417b0b5f-25d5-4007-bdad-91493886b9d9-kube-api-access-s6tnz\") pod \"payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv\" (UID: \"417b0b5f-25d5-4007-bdad-91493886b9d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" Sep 06 06:11:23.703007 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.702990 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97gms\" (UniqueName: \"kubernetes.io/projected/b69a40a3-85f5-4c78-b9ca-8f8980025811-kube-api-access-97gms\") pod \"payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr\" (UID: \"b69a40a3-85f5-4c78-b9ca-8f8980025811\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" Sep 06 06:11:23.805536 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.805470 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" Sep 06 06:11:23.852424 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:23.852396 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" Sep 06 06:11:24.081635 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:24.081609 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv"] Sep 06 06:11:24.083258 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:11:24.083231 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417b0b5f_25d5_4007_bdad_91493886b9d9.slice/crio-a894d52855e313d12818b5c9b70f14ebacd529895fde0095dfa0582bcfad9433 WatchSource:0}: Error finding container a894d52855e313d12818b5c9b70f14ebacd529895fde0095dfa0582bcfad9433: Status 404 returned error can't find the container with id a894d52855e313d12818b5c9b70f14ebacd529895fde0095dfa0582bcfad9433 Sep 06 06:11:24.138967 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:24.138946 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr"] Sep 06 06:11:24.140213 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:11:24.140186 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69a40a3_85f5_4c78_b9ca_8f8980025811.slice/crio-c7e60595f6fc6abf67ba847554da3628c79a0537504a91c8f7d1b67c0f4b3bee WatchSource:0}: Error finding container c7e60595f6fc6abf67ba847554da3628c79a0537504a91c8f7d1b67c0f4b3bee: Status 404 returned error can't find the container with id c7e60595f6fc6abf67ba847554da3628c79a0537504a91c8f7d1b67c0f4b3bee Sep 06 06:11:24.230732 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:24.230694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" event={"ID":"417b0b5f-25d5-4007-bdad-91493886b9d9","Type":"ContainerStarted","Data":"b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38"} Sep 06 06:11:24.230886 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:24.230757 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" event={"ID":"417b0b5f-25d5-4007-bdad-91493886b9d9","Type":"ContainerStarted","Data":"a894d52855e313d12818b5c9b70f14ebacd529895fde0095dfa0582bcfad9433"} Sep 06 06:11:24.230886 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:24.230786 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" Sep 06 06:11:24.232524 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:24.232497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" event={"ID":"b69a40a3-85f5-4c78-b9ca-8f8980025811","Type":"ContainerStarted","Data":"08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f"} Sep 06 06:11:24.232692 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:24.232546 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" event={"ID":"b69a40a3-85f5-4c78-b9ca-8f8980025811","Type":"ContainerStarted","Data":"c7e60595f6fc6abf67ba847554da3628c79a0537504a91c8f7d1b67c0f4b3bee"} Sep 06 06:11:24.232692 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:24.232573 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" Sep 06 06:11:24.248370 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:24.248331 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" podStartSLOduration=1.248318166 podStartE2EDuration="1.248318166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:11:24.24717019 +0000 UTC m=+1049.895789371" watchObservedRunningTime="2026-09-06 06:11:24.248318166 +0000 UTC m=+1049.896937347" Sep 06 06:11:24.262647 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:24.262606 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" podStartSLOduration=1.26259343 podStartE2EDuration="1.26259343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:11:24.262386961 +0000 UTC m=+1049.911006144" watchObservedRunningTime="2026-09-06 06:11:24.26259343 +0000 UTC m=+1049.911212634" Sep 06 06:11:35.240592 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:35.240561 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" Sep 06 06:11:35.241021 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:35.240635 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" Sep 06 06:11:42.904947 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:42.904908 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t"] Sep 06 06:11:42.905406 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:42.905164 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" podUID="a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347" containerName="payload-processing" containerID="cri-o://333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3" gracePeriod=30 Sep 06 06:11:42.914003 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:42.912615 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk"] Sep 06 06:11:42.914003 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:42.912872 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" podUID="21ff4329-7804-483e-b319-b9dbf3b2123d" containerName="payload-pre-processing" containerID="cri-o://5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700" gracePeriod=30 Sep 06 06:11:43.193684 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.193655 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" Sep 06 06:11:43.198442 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.198412 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" Sep 06 06:11:43.276359 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.276328 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phjtl\" (UniqueName: \"kubernetes.io/projected/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347-kube-api-access-phjtl\") pod \"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347\" (UID: \"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347\") " Sep 06 06:11:43.276560 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.276374 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ff4329-7804-483e-b319-b9dbf3b2123d-plugins-config-volume\") pod \"21ff4329-7804-483e-b319-b9dbf3b2123d\" (UID: \"21ff4329-7804-483e-b319-b9dbf3b2123d\") " Sep 06 06:11:43.276560 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.276439 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj6dz\" (UniqueName: \"kubernetes.io/projected/21ff4329-7804-483e-b319-b9dbf3b2123d-kube-api-access-dj6dz\") pod \"21ff4329-7804-483e-b319-b9dbf3b2123d\" (UID: \"21ff4329-7804-483e-b319-b9dbf3b2123d\") " Sep 06 06:11:43.276560 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.276472 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347-plugins-config-volume\") pod \"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347\" (UID: \"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347\") " Sep 06 06:11:43.276889 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.276851 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347" (UID: "a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:11:43.276889 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.276854 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ff4329-7804-483e-b319-b9dbf3b2123d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21ff4329-7804-483e-b319-b9dbf3b2123d" (UID: "21ff4329-7804-483e-b319-b9dbf3b2123d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:11:43.278782 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.278757 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347-kube-api-access-phjtl" (OuterVolumeSpecName: "kube-api-access-phjtl") pod "a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347" (UID: "a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347"). InnerVolumeSpecName "kube-api-access-phjtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:11:43.278889 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.278790 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ff4329-7804-483e-b319-b9dbf3b2123d-kube-api-access-dj6dz" (OuterVolumeSpecName: "kube-api-access-dj6dz") pod "21ff4329-7804-483e-b319-b9dbf3b2123d" (UID: "21ff4329-7804-483e-b319-b9dbf3b2123d"). InnerVolumeSpecName "kube-api-access-dj6dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:11:43.331064 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.331023 2580 generic.go:356] "Generic (PLEG): container finished" podID="a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347" containerID="333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3" exitCode=0 Sep 06 06:11:43.331244 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.331071 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" Sep 06 06:11:43.331244 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.331103 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" event={"ID":"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347","Type":"ContainerDied","Data":"333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3"} Sep 06 06:11:43.331244 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.331162 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t" event={"ID":"a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347","Type":"ContainerDied","Data":"8ac315a327ed3f8ce047710f39042ba8488ea7ac848e8174b790380808b61d89"} Sep 06 06:11:43.331244 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.331196 2580 scope.go:117] "RemoveContainer" containerID="333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3" Sep 06 06:11:43.332457 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.332434 2580 generic.go:356] "Generic (PLEG): container finished" podID="21ff4329-7804-483e-b319-b9dbf3b2123d" containerID="5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700" exitCode=0 Sep 06 06:11:43.332537 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.332473 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" Sep 06 06:11:43.332586 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.332473 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" event={"ID":"21ff4329-7804-483e-b319-b9dbf3b2123d","Type":"ContainerDied","Data":"5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700"} Sep 06 06:11:43.332622 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.332590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk" event={"ID":"21ff4329-7804-483e-b319-b9dbf3b2123d","Type":"ContainerDied","Data":"1254dbd50461859c3f9a643b163ecc99e4e03f87f0c0e4cdf6513b5f7bbaba34"} Sep 06 06:11:43.341409 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.341392 2580 scope.go:117] "RemoveContainer" containerID="333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3" Sep 06 06:11:43.341673 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:11:43.341656 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3\": container with ID starting with 333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3 not found: ID does not exist" containerID="333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3" Sep 06 06:11:43.341752 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.341681 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3"} err="failed to get container status \"333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3\": rpc error: code = NotFound desc = could not find container \"333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3\": container with ID starting with 333b0c1b0b3e7d9a92b5bc40893a5239f1fac8b559c89fb39b7ad1a6d3caacd3 not found: ID does not exist" Sep 06 06:11:43.341752 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.341696 2580 scope.go:117] "RemoveContainer" containerID="5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700" Sep 06 06:11:43.349788 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.349767 2580 scope.go:117] "RemoveContainer" containerID="5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700" Sep 06 06:11:43.350075 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:11:43.350056 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700\": container with ID starting with 5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700 not found: ID does not exist" containerID="5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700" Sep 06 06:11:43.350132 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.350082 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700"} err="failed to get container status \"5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700\": rpc error: code = NotFound desc = could not find container \"5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700\": container with ID starting with 5b01ac5ea5fac3f849184fef7b1ca1bbf13d6e3b4fdec4f6f032d4e3edb6e700 not found: ID does not exist" Sep 06 06:11:43.363319 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.363291 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk"] Sep 06 06:11:43.367059 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.367037 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5da01692-69487988c8-28lbk"] Sep 06 06:11:43.377237 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.377212 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:11:43.377237 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.377233 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phjtl\" (UniqueName: \"kubernetes.io/projected/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347-kube-api-access-phjtl\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:11:43.377376 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.377244 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ff4329-7804-483e-b319-b9dbf3b2123d-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:11:43.377376 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.377254 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj6dz\" (UniqueName: \"kubernetes.io/projected/21ff4329-7804-483e-b319-b9dbf3b2123d-kube-api-access-dj6dz\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:11:43.377793 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.377773 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t"] Sep 06 06:11:43.381214 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:43.381193 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5da01692-7b5594b68c-f769t"] Sep 06 06:11:44.875785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:44.875749 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21ff4329-7804-483e-b319-b9dbf3b2123d" path="/var/lib/kubelet/pods/21ff4329-7804-483e-b319-b9dbf3b2123d/volumes" Sep 06 06:11:44.876178 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:44.876140 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347" path="/var/lib/kubelet/pods/a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347/volumes" Sep 06 06:11:47.396595 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.396501 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd"] Sep 06 06:11:47.397086 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.397020 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ff4329-7804-483e-b319-b9dbf3b2123d" containerName="payload-pre-processing" Sep 06 06:11:47.397086 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.397037 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ff4329-7804-483e-b319-b9dbf3b2123d" containerName="payload-pre-processing" Sep 06 06:11:47.397086 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.397054 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347" containerName="payload-processing" Sep 06 06:11:47.397086 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.397060 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347" containerName="payload-processing" Sep 06 06:11:47.397298 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.397123 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ff4329-7804-483e-b319-b9dbf3b2123d" containerName="payload-pre-processing" Sep 06 06:11:47.397298 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.397132 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d2bac2-9b87-4cfc-8d13-e40ccbbb9347" containerName="payload-processing" Sep 06 06:11:47.400427 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.400403 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" Sep 06 06:11:47.403337 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.403313 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-06e64f94-dockercfg-585mg\"" Sep 06 06:11:47.403553 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.403539 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-06e64f94\"" Sep 06 06:11:47.411321 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.411267 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd"] Sep 06 06:11:47.422402 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.421584 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh"] Sep 06 06:11:47.426306 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.426279 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" Sep 06 06:11:47.437596 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.437572 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh"] Sep 06 06:11:47.517051 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.517010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8a254c-f833-4ffd-beb0-9a3ebaf38831-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd\" (UID: \"ae8a254c-f833-4ffd-beb0-9a3ebaf38831\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" Sep 06 06:11:47.517051 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.517055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wbg7\" (UniqueName: \"kubernetes.io/projected/ae8a254c-f833-4ffd-beb0-9a3ebaf38831-kube-api-access-2wbg7\") pod \"payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd\" (UID: \"ae8a254c-f833-4ffd-beb0-9a3ebaf38831\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" Sep 06 06:11:47.517295 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.517075 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbkjn\" (UniqueName: \"kubernetes.io/projected/c17a9b19-dd51-4f4a-9b1f-de753f6592b6-kube-api-access-qbkjn\") pod \"payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh\" (UID: \"c17a9b19-dd51-4f4a-9b1f-de753f6592b6\") " pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" Sep 06 06:11:47.517295 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.517176 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17a9b19-dd51-4f4a-9b1f-de753f6592b6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh\" (UID: \"c17a9b19-dd51-4f4a-9b1f-de753f6592b6\") " pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" Sep 06 06:11:47.618022 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.617986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8a254c-f833-4ffd-beb0-9a3ebaf38831-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd\" (UID: \"ae8a254c-f833-4ffd-beb0-9a3ebaf38831\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" Sep 06 06:11:47.618191 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.618033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wbg7\" (UniqueName: \"kubernetes.io/projected/ae8a254c-f833-4ffd-beb0-9a3ebaf38831-kube-api-access-2wbg7\") pod \"payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd\" (UID: \"ae8a254c-f833-4ffd-beb0-9a3ebaf38831\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" Sep 06 06:11:47.618191 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.618056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbkjn\" (UniqueName: \"kubernetes.io/projected/c17a9b19-dd51-4f4a-9b1f-de753f6592b6-kube-api-access-qbkjn\") pod \"payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh\" (UID: \"c17a9b19-dd51-4f4a-9b1f-de753f6592b6\") " pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" Sep 06 06:11:47.618297 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.618240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17a9b19-dd51-4f4a-9b1f-de753f6592b6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh\" (UID: \"c17a9b19-dd51-4f4a-9b1f-de753f6592b6\") " pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" Sep 06 06:11:47.618781 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.618764 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8a254c-f833-4ffd-beb0-9a3ebaf38831-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd\" (UID: \"ae8a254c-f833-4ffd-beb0-9a3ebaf38831\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" Sep 06 06:11:47.619155 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.619132 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17a9b19-dd51-4f4a-9b1f-de753f6592b6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh\" (UID: \"c17a9b19-dd51-4f4a-9b1f-de753f6592b6\") " pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" Sep 06 06:11:47.627390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.627359 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wbg7\" (UniqueName: \"kubernetes.io/projected/ae8a254c-f833-4ffd-beb0-9a3ebaf38831-kube-api-access-2wbg7\") pod \"payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd\" (UID: \"ae8a254c-f833-4ffd-beb0-9a3ebaf38831\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" Sep 06 06:11:47.627706 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.627689 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbkjn\" (UniqueName: \"kubernetes.io/projected/c17a9b19-dd51-4f4a-9b1f-de753f6592b6-kube-api-access-qbkjn\") pod \"payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh\" (UID: \"c17a9b19-dd51-4f4a-9b1f-de753f6592b6\") " pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" Sep 06 06:11:47.712167 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.712127 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" Sep 06 06:11:47.739934 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.739899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" Sep 06 06:11:47.991211 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:47.991180 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd"] Sep 06 06:11:47.992663 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:11:47.992638 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8a254c_f833_4ffd_beb0_9a3ebaf38831.slice/crio-38909d336448130c2ef6ef00e17a97b282633f75c1014d505a6dead8a3e0dbf2 WatchSource:0}: Error finding container 38909d336448130c2ef6ef00e17a97b282633f75c1014d505a6dead8a3e0dbf2: Status 404 returned error can't find the container with id 38909d336448130c2ef6ef00e17a97b282633f75c1014d505a6dead8a3e0dbf2 Sep 06 06:11:48.023215 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:48.023190 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh"] Sep 06 06:11:48.024237 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:11:48.024215 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17a9b19_dd51_4f4a_9b1f_de753f6592b6.slice/crio-367bbed565ce15b756960ef12ab1a9da7834965e3899722e7b10e74ca61c51ea WatchSource:0}: Error finding container 367bbed565ce15b756960ef12ab1a9da7834965e3899722e7b10e74ca61c51ea: Status 404 returned error can't find the container with id 367bbed565ce15b756960ef12ab1a9da7834965e3899722e7b10e74ca61c51ea Sep 06 06:11:48.363715 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:48.363601 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" event={"ID":"c17a9b19-dd51-4f4a-9b1f-de753f6592b6","Type":"ContainerStarted","Data":"6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741"} Sep 06 06:11:48.363715 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:48.363674 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" event={"ID":"c17a9b19-dd51-4f4a-9b1f-de753f6592b6","Type":"ContainerStarted","Data":"367bbed565ce15b756960ef12ab1a9da7834965e3899722e7b10e74ca61c51ea"} Sep 06 06:11:48.363715 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:48.363707 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" Sep 06 06:11:48.365323 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:48.365294 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" event={"ID":"ae8a254c-f833-4ffd-beb0-9a3ebaf38831","Type":"ContainerStarted","Data":"c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0"} Sep 06 06:11:48.365422 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:48.365349 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" Sep 06 06:11:48.365422 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:48.365370 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" event={"ID":"ae8a254c-f833-4ffd-beb0-9a3ebaf38831","Type":"ContainerStarted","Data":"38909d336448130c2ef6ef00e17a97b282633f75c1014d505a6dead8a3e0dbf2"} Sep 06 06:11:48.385358 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:48.385307 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" podStartSLOduration=1.385292231 podStartE2EDuration="1.385292231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:11:48.383946456 +0000 UTC m=+1074.032565638" watchObservedRunningTime="2026-09-06 06:11:48.385292231 +0000 UTC m=+1074.033911403" Sep 06 06:11:48.403181 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:48.403130 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" podStartSLOduration=1.403118109 podStartE2EDuration="1.403118109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:11:48.401529347 +0000 UTC m=+1074.050148545" watchObservedRunningTime="2026-09-06 06:11:48.403118109 +0000 UTC m=+1074.051737285" Sep 06 06:11:59.373022 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:59.372992 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" Sep 06 06:11:59.373394 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:11:59.373061 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" Sep 06 06:12:09.890688 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:09.890650 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz"] Sep 06 06:12:09.895944 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:09.895921 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:09.898666 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:09.898639 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-d5939f9b-kserve-self-signed-certs\"" Sep 06 06:12:09.904169 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:09.904148 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz"] Sep 06 06:12:09.917673 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:09.917639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-dshm\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:09.917847 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:09.917689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-model-cache\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:09.917847 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:09.917766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfnr6\" (UniqueName: \"kubernetes.io/projected/1cce4976-e9bf-4d16-98a1-c574323ff5e2-kube-api-access-wfnr6\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:09.917981 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:09.917892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cce4976-e9bf-4d16-98a1-c574323ff5e2-tls-certs\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:09.917981 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:09.917930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-home\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:09.917981 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:09.917955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-tmp-dir\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.019070 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.019023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cce4976-e9bf-4d16-98a1-c574323ff5e2-tls-certs\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.019070 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.019073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-home\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.019349 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.019090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-tmp-dir\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.019349 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.019137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-dshm\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.019349 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.019164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-model-cache\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.019349 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.019306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfnr6\" (UniqueName: \"kubernetes.io/projected/1cce4976-e9bf-4d16-98a1-c574323ff5e2-kube-api-access-wfnr6\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.019561 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.019520 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-home\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.019561 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.019536 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-model-cache\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.019648 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.019627 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-tmp-dir\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.021495 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.021469 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-dshm\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.021747 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.021729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cce4976-e9bf-4d16-98a1-c574323ff5e2-tls-certs\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.027168 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.027149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfnr6\" (UniqueName: \"kubernetes.io/projected/1cce4976-e9bf-4d16-98a1-c574323ff5e2-kube-api-access-wfnr6\") pod \"e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.206789 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.206755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:10.478369 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.478339 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz"] Sep 06 06:12:10.480301 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:12:10.480273 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cce4976_e9bf_4d16_98a1_c574323ff5e2.slice/crio-fe496a59b45ef0be87866b2e1405d0c06fd633efdbc2f232c71f86ebdd639bd4 WatchSource:0}: Error finding container fe496a59b45ef0be87866b2e1405d0c06fd633efdbc2f232c71f86ebdd639bd4: Status 404 returned error can't find the container with id fe496a59b45ef0be87866b2e1405d0c06fd633efdbc2f232c71f86ebdd639bd4 Sep 06 06:12:10.489252 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:10.489225 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" event={"ID":"1cce4976-e9bf-4d16-98a1-c574323ff5e2","Type":"ContainerStarted","Data":"fe496a59b45ef0be87866b2e1405d0c06fd633efdbc2f232c71f86ebdd639bd4"} Sep 06 06:12:11.496309 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:11.496261 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" event={"ID":"1cce4976-e9bf-4d16-98a1-c574323ff5e2","Type":"ContainerStarted","Data":"1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6"} Sep 06 06:12:11.513003 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:11.512946 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" podStartSLOduration=2.512929807 podStartE2EDuration="2.512929807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:12:11.510506267 +0000 UTC m=+1097.159125447" watchObservedRunningTime="2026-09-06 06:12:11.512929807 +0000 UTC m=+1097.161548986" Sep 06 06:12:17.979499 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:17.979464 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh"] Sep 06 06:12:17.980005 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:17.979702 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" podUID="c17a9b19-dd51-4f4a-9b1f-de753f6592b6" containerName="payload-processing" containerID="cri-o://6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741" gracePeriod=30 Sep 06 06:12:17.983738 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:17.983700 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd"] Sep 06 06:12:17.984902 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:17.984708 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" podUID="ae8a254c-f833-4ffd-beb0-9a3ebaf38831" containerName="payload-pre-processing" containerID="cri-o://c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0" gracePeriod=30 Sep 06 06:12:18.281797 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.281771 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" Sep 06 06:12:18.285078 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.285048 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" Sep 06 06:12:18.400668 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.400626 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8a254c-f833-4ffd-beb0-9a3ebaf38831-plugins-config-volume\") pod \"ae8a254c-f833-4ffd-beb0-9a3ebaf38831\" (UID: \"ae8a254c-f833-4ffd-beb0-9a3ebaf38831\") " Sep 06 06:12:18.400879 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.400685 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17a9b19-dd51-4f4a-9b1f-de753f6592b6-plugins-config-volume\") pod \"c17a9b19-dd51-4f4a-9b1f-de753f6592b6\" (UID: \"c17a9b19-dd51-4f4a-9b1f-de753f6592b6\") " Sep 06 06:12:18.400879 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.400703 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbkjn\" (UniqueName: \"kubernetes.io/projected/c17a9b19-dd51-4f4a-9b1f-de753f6592b6-kube-api-access-qbkjn\") pod \"c17a9b19-dd51-4f4a-9b1f-de753f6592b6\" (UID: \"c17a9b19-dd51-4f4a-9b1f-de753f6592b6\") " Sep 06 06:12:18.400879 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.400744 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wbg7\" (UniqueName: \"kubernetes.io/projected/ae8a254c-f833-4ffd-beb0-9a3ebaf38831-kube-api-access-2wbg7\") pod \"ae8a254c-f833-4ffd-beb0-9a3ebaf38831\" (UID: \"ae8a254c-f833-4ffd-beb0-9a3ebaf38831\") " Sep 06 06:12:18.401153 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.401128 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8a254c-f833-4ffd-beb0-9a3ebaf38831-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae8a254c-f833-4ffd-beb0-9a3ebaf38831" (UID: "ae8a254c-f833-4ffd-beb0-9a3ebaf38831"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:12:18.401208 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.401144 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17a9b19-dd51-4f4a-9b1f-de753f6592b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c17a9b19-dd51-4f4a-9b1f-de753f6592b6" (UID: "c17a9b19-dd51-4f4a-9b1f-de753f6592b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:12:18.402858 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.402823 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17a9b19-dd51-4f4a-9b1f-de753f6592b6-kube-api-access-qbkjn" (OuterVolumeSpecName: "kube-api-access-qbkjn") pod "c17a9b19-dd51-4f4a-9b1f-de753f6592b6" (UID: "c17a9b19-dd51-4f4a-9b1f-de753f6592b6"). InnerVolumeSpecName "kube-api-access-qbkjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:12:18.402930 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.402865 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8a254c-f833-4ffd-beb0-9a3ebaf38831-kube-api-access-2wbg7" (OuterVolumeSpecName: "kube-api-access-2wbg7") pod "ae8a254c-f833-4ffd-beb0-9a3ebaf38831" (UID: "ae8a254c-f833-4ffd-beb0-9a3ebaf38831"). InnerVolumeSpecName "kube-api-access-2wbg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:12:18.502201 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.502107 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17a9b19-dd51-4f4a-9b1f-de753f6592b6-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:12:18.502201 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.502140 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbkjn\" (UniqueName: \"kubernetes.io/projected/c17a9b19-dd51-4f4a-9b1f-de753f6592b6-kube-api-access-qbkjn\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:12:18.502201 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.502152 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wbg7\" (UniqueName: \"kubernetes.io/projected/ae8a254c-f833-4ffd-beb0-9a3ebaf38831-kube-api-access-2wbg7\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:12:18.502201 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.502160 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8a254c-f833-4ffd-beb0-9a3ebaf38831-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:12:18.535167 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.535131 2580 generic.go:356] "Generic (PLEG): container finished" podID="c17a9b19-dd51-4f4a-9b1f-de753f6592b6" containerID="6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741" exitCode=0 Sep 06 06:12:18.535339 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.535184 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" Sep 06 06:12:18.535339 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.535214 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" event={"ID":"c17a9b19-dd51-4f4a-9b1f-de753f6592b6","Type":"ContainerDied","Data":"6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741"} Sep 06 06:12:18.535339 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.535263 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh" event={"ID":"c17a9b19-dd51-4f4a-9b1f-de753f6592b6","Type":"ContainerDied","Data":"367bbed565ce15b756960ef12ab1a9da7834965e3899722e7b10e74ca61c51ea"} Sep 06 06:12:18.535339 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.535285 2580 scope.go:117] "RemoveContainer" containerID="6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741" Sep 06 06:12:18.536691 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.536667 2580 generic.go:356] "Generic (PLEG): container finished" podID="ae8a254c-f833-4ffd-beb0-9a3ebaf38831" containerID="c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0" exitCode=0 Sep 06 06:12:18.536823 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.536692 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" event={"ID":"ae8a254c-f833-4ffd-beb0-9a3ebaf38831","Type":"ContainerDied","Data":"c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0"} Sep 06 06:12:18.536823 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.536718 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" Sep 06 06:12:18.536823 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.536734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd" event={"ID":"ae8a254c-f833-4ffd-beb0-9a3ebaf38831","Type":"ContainerDied","Data":"38909d336448130c2ef6ef00e17a97b282633f75c1014d505a6dead8a3e0dbf2"} Sep 06 06:12:18.546847 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.546747 2580 scope.go:117] "RemoveContainer" containerID="6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741" Sep 06 06:12:18.547128 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:12:18.547109 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741\": container with ID starting with 6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741 not found: ID does not exist" containerID="6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741" Sep 06 06:12:18.547202 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.547139 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741"} err="failed to get container status \"6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741\": rpc error: code = NotFound desc = could not find container \"6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741\": container with ID starting with 6bd8c2fecaf099cc6c28b5ececaa4d0e5ee773bf2fe986c58f97073e6c214741 not found: ID does not exist" Sep 06 06:12:18.547202 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.547160 2580 scope.go:117] "RemoveContainer" containerID="c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0" Sep 06 06:12:18.555625 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.555602 2580 scope.go:117] "RemoveContainer" containerID="c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0" Sep 06 06:12:18.556011 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:12:18.555979 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0\": container with ID starting with c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0 not found: ID does not exist" containerID="c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0" Sep 06 06:12:18.556114 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.556021 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0"} err="failed to get container status \"c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0\": rpc error: code = NotFound desc = could not find container \"c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0\": container with ID starting with c47bf3341bfa2d7bed8ed27b03000fcb3db1dd8aefb327401578d4c7e746fad0 not found: ID does not exist" Sep 06 06:12:18.557956 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.557936 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh"] Sep 06 06:12:18.562855 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.562820 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-06e64f94-85645ff8c4-lq8fh"] Sep 06 06:12:18.573283 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.573257 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd"] Sep 06 06:12:18.576749 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.576727 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-06e64f94-59f5f5fd6b-hmsrd"] Sep 06 06:12:18.875799 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.875700 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae8a254c-f833-4ffd-beb0-9a3ebaf38831" path="/var/lib/kubelet/pods/ae8a254c-f833-4ffd-beb0-9a3ebaf38831/volumes" Sep 06 06:12:18.876249 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:18.876228 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c17a9b19-dd51-4f4a-9b1f-de753f6592b6" path="/var/lib/kubelet/pods/c17a9b19-dd51-4f4a-9b1f-de753f6592b6/volumes" Sep 06 06:12:20.207793 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:20.207751 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:20.208215 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:20.207864 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:20.220474 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:20.220444 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:20.559530 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:20.559450 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:12:23.170708 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.170673 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf"] Sep 06 06:12:23.171125 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.171112 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8a254c-f833-4ffd-beb0-9a3ebaf38831" containerName="payload-pre-processing" Sep 06 06:12:23.171169 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.171127 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8a254c-f833-4ffd-beb0-9a3ebaf38831" containerName="payload-pre-processing" Sep 06 06:12:23.171169 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.171137 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c17a9b19-dd51-4f4a-9b1f-de753f6592b6" containerName="payload-processing" Sep 06 06:12:23.171169 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.171142 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c17a9b19-dd51-4f4a-9b1f-de753f6592b6" containerName="payload-processing" Sep 06 06:12:23.171268 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.171210 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c17a9b19-dd51-4f4a-9b1f-de753f6592b6" containerName="payload-processing" Sep 06 06:12:23.171268 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.171223 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8a254c-f833-4ffd-beb0-9a3ebaf38831" containerName="payload-pre-processing" Sep 06 06:12:23.174434 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.174413 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" Sep 06 06:12:23.177292 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.177266 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-efdea9ac\"" Sep 06 06:12:23.177433 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.177414 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-efdea9ac-dockercfg-prccc\"" Sep 06 06:12:23.186127 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.186101 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf"] Sep 06 06:12:23.207665 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.207637 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb"] Sep 06 06:12:23.211125 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.211103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" Sep 06 06:12:23.228262 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.228236 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb"] Sep 06 06:12:23.245760 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.245730 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da241295-d0e5-40cd-9c68-e5ec680c66a9-plugins-config-volume\") pod \"payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb\" (UID: \"da241295-d0e5-40cd-9c68-e5ec680c66a9\") " pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" Sep 06 06:12:23.245894 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.245800 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7mlw\" (UniqueName: \"kubernetes.io/projected/da241295-d0e5-40cd-9c68-e5ec680c66a9-kube-api-access-m7mlw\") pod \"payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb\" (UID: \"da241295-d0e5-40cd-9c68-e5ec680c66a9\") " pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" Sep 06 06:12:23.245894 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.245824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf\" (UID: \"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" Sep 06 06:12:23.245997 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.245937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-228zl\" (UniqueName: \"kubernetes.io/projected/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1-kube-api-access-228zl\") pod \"payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf\" (UID: \"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" Sep 06 06:12:23.347244 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.347206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-228zl\" (UniqueName: \"kubernetes.io/projected/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1-kube-api-access-228zl\") pod \"payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf\" (UID: \"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" Sep 06 06:12:23.347421 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.347273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da241295-d0e5-40cd-9c68-e5ec680c66a9-plugins-config-volume\") pod \"payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb\" (UID: \"da241295-d0e5-40cd-9c68-e5ec680c66a9\") " pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" Sep 06 06:12:23.347421 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.347323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7mlw\" (UniqueName: \"kubernetes.io/projected/da241295-d0e5-40cd-9c68-e5ec680c66a9-kube-api-access-m7mlw\") pod \"payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb\" (UID: \"da241295-d0e5-40cd-9c68-e5ec680c66a9\") " pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" Sep 06 06:12:23.347421 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.347340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf\" (UID: \"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" Sep 06 06:12:23.348083 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.348065 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf\" (UID: \"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" Sep 06 06:12:23.348131 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.348067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da241295-d0e5-40cd-9c68-e5ec680c66a9-plugins-config-volume\") pod \"payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb\" (UID: \"da241295-d0e5-40cd-9c68-e5ec680c66a9\") " pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" Sep 06 06:12:23.355191 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.355158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-228zl\" (UniqueName: \"kubernetes.io/projected/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1-kube-api-access-228zl\") pod \"payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf\" (UID: \"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" Sep 06 06:12:23.355299 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.355205 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7mlw\" (UniqueName: \"kubernetes.io/projected/da241295-d0e5-40cd-9c68-e5ec680c66a9-kube-api-access-m7mlw\") pod \"payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb\" (UID: \"da241295-d0e5-40cd-9c68-e5ec680c66a9\") " pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" Sep 06 06:12:23.486246 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.486153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" Sep 06 06:12:23.521078 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.521048 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" Sep 06 06:12:23.766145 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.766121 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf"] Sep 06 06:12:23.768370 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:12:23.768347 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22e8cc1_baa6_4fcf_a9fd_e9f311cc4af1.slice/crio-05bcdd8216fde8f3092db2499eb3c38be09cbd2809fae788834eda0dfaf0028a WatchSource:0}: Error finding container 05bcdd8216fde8f3092db2499eb3c38be09cbd2809fae788834eda0dfaf0028a: Status 404 returned error can't find the container with id 05bcdd8216fde8f3092db2499eb3c38be09cbd2809fae788834eda0dfaf0028a Sep 06 06:12:23.805294 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:23.805269 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb"] Sep 06 06:12:23.819028 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:12:23.819000 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda241295_d0e5_40cd_9c68_e5ec680c66a9.slice/crio-cfbede5460ba6f60672a9e0931ac81902369bbea4b7e6b3a0eecef3676cce6b2 WatchSource:0}: Error finding container cfbede5460ba6f60672a9e0931ac81902369bbea4b7e6b3a0eecef3676cce6b2: Status 404 returned error can't find the container with id cfbede5460ba6f60672a9e0931ac81902369bbea4b7e6b3a0eecef3676cce6b2 Sep 06 06:12:24.574783 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:24.574739 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" event={"ID":"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1","Type":"ContainerStarted","Data":"11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080"} Sep 06 06:12:24.575294 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:24.574797 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" event={"ID":"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1","Type":"ContainerStarted","Data":"05bcdd8216fde8f3092db2499eb3c38be09cbd2809fae788834eda0dfaf0028a"} Sep 06 06:12:24.575294 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:24.574861 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" Sep 06 06:12:24.576502 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:24.576473 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" event={"ID":"da241295-d0e5-40cd-9c68-e5ec680c66a9","Type":"ContainerStarted","Data":"ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9"} Sep 06 06:12:24.576645 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:24.576519 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" event={"ID":"da241295-d0e5-40cd-9c68-e5ec680c66a9","Type":"ContainerStarted","Data":"cfbede5460ba6f60672a9e0931ac81902369bbea4b7e6b3a0eecef3676cce6b2"} Sep 06 06:12:24.576645 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:24.576564 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" Sep 06 06:12:24.590563 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:24.590499 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" podStartSLOduration=1.5904843789999998 podStartE2EDuration="1.590484379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:12:24.589733219 +0000 UTC m=+1110.238352399" watchObservedRunningTime="2026-09-06 06:12:24.590484379 +0000 UTC m=+1110.239103561" Sep 06 06:12:24.607840 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:24.607784 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" podStartSLOduration=1.607768501 podStartE2EDuration="1.607768501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:12:24.605544654 +0000 UTC m=+1110.254163833" watchObservedRunningTime="2026-09-06 06:12:24.607768501 +0000 UTC m=+1110.256387726" Sep 06 06:12:35.583984 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:35.583953 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" Sep 06 06:12:35.584376 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:35.584028 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" Sep 06 06:12:53.263861 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.263803 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb"] Sep 06 06:12:53.264395 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.264103 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" podUID="da241295-d0e5-40cd-9c68-e5ec680c66a9" containerName="payload-processing" containerID="cri-o://ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9" gracePeriod=30 Sep 06 06:12:53.274169 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.274132 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf"] Sep 06 06:12:53.274429 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.274397 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" podUID="f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1" containerName="payload-pre-processing" containerID="cri-o://11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080" gracePeriod=30 Sep 06 06:12:53.546415 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.546391 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" Sep 06 06:12:53.549423 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.549401 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" Sep 06 06:12:53.648356 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.648318 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da241295-d0e5-40cd-9c68-e5ec680c66a9-plugins-config-volume\") pod \"da241295-d0e5-40cd-9c68-e5ec680c66a9\" (UID: \"da241295-d0e5-40cd-9c68-e5ec680c66a9\") " Sep 06 06:12:53.648545 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.648373 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1-plugins-config-volume\") pod \"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1\" (UID: \"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1\") " Sep 06 06:12:53.648545 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.648406 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-228zl\" (UniqueName: \"kubernetes.io/projected/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1-kube-api-access-228zl\") pod \"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1\" (UID: \"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1\") " Sep 06 06:12:53.648545 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.648425 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7mlw\" (UniqueName: \"kubernetes.io/projected/da241295-d0e5-40cd-9c68-e5ec680c66a9-kube-api-access-m7mlw\") pod \"da241295-d0e5-40cd-9c68-e5ec680c66a9\" (UID: \"da241295-d0e5-40cd-9c68-e5ec680c66a9\") " Sep 06 06:12:53.648776 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.648746 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da241295-d0e5-40cd-9c68-e5ec680c66a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da241295-d0e5-40cd-9c68-e5ec680c66a9" (UID: "da241295-d0e5-40cd-9c68-e5ec680c66a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:12:53.648983 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.648773 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1" (UID: "f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:12:53.650687 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.650654 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1-kube-api-access-228zl" (OuterVolumeSpecName: "kube-api-access-228zl") pod "f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1" (UID: "f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1"). InnerVolumeSpecName "kube-api-access-228zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:12:53.650800 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.650776 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da241295-d0e5-40cd-9c68-e5ec680c66a9-kube-api-access-m7mlw" (OuterVolumeSpecName: "kube-api-access-m7mlw") pod "da241295-d0e5-40cd-9c68-e5ec680c66a9" (UID: "da241295-d0e5-40cd-9c68-e5ec680c66a9"). InnerVolumeSpecName "kube-api-access-m7mlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:12:53.728914 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.728880 2580 generic.go:356] "Generic (PLEG): container finished" podID="f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1" containerID="11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080" exitCode=0 Sep 06 06:12:53.729099 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.728929 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" Sep 06 06:12:53.729099 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.728971 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" event={"ID":"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1","Type":"ContainerDied","Data":"11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080"} Sep 06 06:12:53.729099 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.729028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf" event={"ID":"f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1","Type":"ContainerDied","Data":"05bcdd8216fde8f3092db2499eb3c38be09cbd2809fae788834eda0dfaf0028a"} Sep 06 06:12:53.729099 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.729057 2580 scope.go:117] "RemoveContainer" containerID="11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080" Sep 06 06:12:53.730573 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.730548 2580 generic.go:356] "Generic (PLEG): container finished" podID="da241295-d0e5-40cd-9c68-e5ec680c66a9" containerID="ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9" exitCode=0 Sep 06 06:12:53.730704 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.730592 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" event={"ID":"da241295-d0e5-40cd-9c68-e5ec680c66a9","Type":"ContainerDied","Data":"ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9"} Sep 06 06:12:53.730704 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.730601 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" Sep 06 06:12:53.730704 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.730638 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb" event={"ID":"da241295-d0e5-40cd-9c68-e5ec680c66a9","Type":"ContainerDied","Data":"cfbede5460ba6f60672a9e0931ac81902369bbea4b7e6b3a0eecef3676cce6b2"} Sep 06 06:12:53.739521 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.739498 2580 scope.go:117] "RemoveContainer" containerID="11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080" Sep 06 06:12:53.739778 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:12:53.739756 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080\": container with ID starting with 11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080 not found: ID does not exist" containerID="11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080" Sep 06 06:12:53.739865 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.739788 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080"} err="failed to get container status \"11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080\": rpc error: code = NotFound desc = could not find container \"11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080\": container with ID starting with 11b95010ca2c70953d5b5cf6d43d5be218f30986b1225bab41d3507154c5c080 not found: ID does not exist" Sep 06 06:12:53.739865 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.739804 2580 scope.go:117] "RemoveContainer" containerID="ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9" Sep 06 06:12:53.749051 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.749027 2580 scope.go:117] "RemoveContainer" containerID="ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9" Sep 06 06:12:53.749362 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:12:53.749338 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9\": container with ID starting with ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9 not found: ID does not exist" containerID="ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9" Sep 06 06:12:53.749453 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.749372 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9"} err="failed to get container status \"ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9\": rpc error: code = NotFound desc = could not find container \"ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9\": container with ID starting with ad62499a2df63d0af89c805b4ce3ed28a2999bc40ba42a6971f54e64de23b1f9 not found: ID does not exist" Sep 06 06:12:53.749453 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.749392 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da241295-d0e5-40cd-9c68-e5ec680c66a9-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:12:53.749453 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.749417 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:12:53.749453 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.749432 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-228zl\" (UniqueName: \"kubernetes.io/projected/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1-kube-api-access-228zl\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:12:53.749453 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.749448 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7mlw\" (UniqueName: \"kubernetes.io/projected/da241295-d0e5-40cd-9c68-e5ec680c66a9-kube-api-access-m7mlw\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:12:53.753683 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.753660 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf"] Sep 06 06:12:53.763208 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.763183 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-efdea9ac-b899477d6-cqghf"] Sep 06 06:12:53.771713 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.771655 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb"] Sep 06 06:12:53.773407 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:53.773390 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-efdea9ac-66d4589995-6h6pb"] Sep 06 06:12:54.876206 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:54.876165 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da241295-d0e5-40cd-9c68-e5ec680c66a9" path="/var/lib/kubelet/pods/da241295-d0e5-40cd-9c68-e5ec680c66a9/volumes" Sep 06 06:12:54.876713 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:54.876693 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1" path="/var/lib/kubelet/pods/f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1/volumes" Sep 06 06:12:59.401812 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.401768 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr"] Sep 06 06:12:59.402314 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.402205 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1" containerName="payload-pre-processing" Sep 06 06:12:59.402314 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.402218 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1" containerName="payload-pre-processing" Sep 06 06:12:59.402314 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.402231 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da241295-d0e5-40cd-9c68-e5ec680c66a9" containerName="payload-processing" Sep 06 06:12:59.402314 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.402236 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da241295-d0e5-40cd-9c68-e5ec680c66a9" containerName="payload-processing" Sep 06 06:12:59.402314 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.402311 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da241295-d0e5-40cd-9c68-e5ec680c66a9" containerName="payload-processing" Sep 06 06:12:59.402583 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.402320 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22e8cc1-baa6-4fcf-a9fd-e9f311cc4af1" containerName="payload-pre-processing" Sep 06 06:12:59.407116 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.407096 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" Sep 06 06:12:59.410009 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.409986 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f260fc83\"" Sep 06 06:12:59.410138 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.410032 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f260fc83-dockercfg-m6n2k\"" Sep 06 06:12:59.414215 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.414194 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr"] Sep 06 06:12:59.437730 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.437699 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk"] Sep 06 06:12:59.441219 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.441202 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" Sep 06 06:12:59.451473 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.451448 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk"] Sep 06 06:12:59.501771 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.501737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872jl\" (UniqueName: \"kubernetes.io/projected/2caba584-fd59-4567-aad7-3287cfd87f21-kube-api-access-872jl\") pod \"payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr\" (UID: \"2caba584-fd59-4567-aad7-3287cfd87f21\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" Sep 06 06:12:59.501932 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.501795 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49aff0de-2f7e-41ec-aadd-d7f38f18a742-plugins-config-volume\") pod \"payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk\" (UID: \"49aff0de-2f7e-41ec-aadd-d7f38f18a742\") " pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" Sep 06 06:12:59.501978 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.501962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vvcg\" (UniqueName: \"kubernetes.io/projected/49aff0de-2f7e-41ec-aadd-d7f38f18a742-kube-api-access-5vvcg\") pod \"payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk\" (UID: \"49aff0de-2f7e-41ec-aadd-d7f38f18a742\") " pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" Sep 06 06:12:59.502019 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.502010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caba584-fd59-4567-aad7-3287cfd87f21-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr\" (UID: \"2caba584-fd59-4567-aad7-3287cfd87f21\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" Sep 06 06:12:59.603129 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.603077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49aff0de-2f7e-41ec-aadd-d7f38f18a742-plugins-config-volume\") pod \"payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk\" (UID: \"49aff0de-2f7e-41ec-aadd-d7f38f18a742\") " pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" Sep 06 06:12:59.603329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.603187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vvcg\" (UniqueName: \"kubernetes.io/projected/49aff0de-2f7e-41ec-aadd-d7f38f18a742-kube-api-access-5vvcg\") pod \"payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk\" (UID: \"49aff0de-2f7e-41ec-aadd-d7f38f18a742\") " pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" Sep 06 06:12:59.603329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.603225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caba584-fd59-4567-aad7-3287cfd87f21-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr\" (UID: \"2caba584-fd59-4567-aad7-3287cfd87f21\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" Sep 06 06:12:59.603329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.603248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872jl\" (UniqueName: \"kubernetes.io/projected/2caba584-fd59-4567-aad7-3287cfd87f21-kube-api-access-872jl\") pod \"payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr\" (UID: \"2caba584-fd59-4567-aad7-3287cfd87f21\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" Sep 06 06:12:59.603804 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.603778 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49aff0de-2f7e-41ec-aadd-d7f38f18a742-plugins-config-volume\") pod \"payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk\" (UID: \"49aff0de-2f7e-41ec-aadd-d7f38f18a742\") " pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" Sep 06 06:12:59.604017 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.603997 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caba584-fd59-4567-aad7-3287cfd87f21-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr\" (UID: \"2caba584-fd59-4567-aad7-3287cfd87f21\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" Sep 06 06:12:59.611695 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.611666 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-872jl\" (UniqueName: \"kubernetes.io/projected/2caba584-fd59-4567-aad7-3287cfd87f21-kube-api-access-872jl\") pod \"payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr\" (UID: \"2caba584-fd59-4567-aad7-3287cfd87f21\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" Sep 06 06:12:59.611782 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.611747 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vvcg\" (UniqueName: \"kubernetes.io/projected/49aff0de-2f7e-41ec-aadd-d7f38f18a742-kube-api-access-5vvcg\") pod \"payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk\" (UID: \"49aff0de-2f7e-41ec-aadd-d7f38f18a742\") " pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" Sep 06 06:12:59.718326 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.718293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" Sep 06 06:12:59.752244 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.752209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" Sep 06 06:12:59.997403 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:12:59.997374 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr"] Sep 06 06:13:00.000118 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:13:00.000090 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2caba584_fd59_4567_aad7_3287cfd87f21.slice/crio-1435a620eab08edac8babedcb04fb8016633fbae70b26e7a670e019dd79d1382 WatchSource:0}: Error finding container 1435a620eab08edac8babedcb04fb8016633fbae70b26e7a670e019dd79d1382: Status 404 returned error can't find the container with id 1435a620eab08edac8babedcb04fb8016633fbae70b26e7a670e019dd79d1382 Sep 06 06:13:00.056171 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:00.056144 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk"] Sep 06 06:13:00.058158 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:13:00.058128 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49aff0de_2f7e_41ec_aadd_d7f38f18a742.slice/crio-0d6ef047851eba69e6735b31fec8502cc873449920c09df377ff7c5bb129185d WatchSource:0}: Error finding container 0d6ef047851eba69e6735b31fec8502cc873449920c09df377ff7c5bb129185d: Status 404 returned error can't find the container with id 0d6ef047851eba69e6735b31fec8502cc873449920c09df377ff7c5bb129185d Sep 06 06:13:00.774643 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:00.774603 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" event={"ID":"2caba584-fd59-4567-aad7-3287cfd87f21","Type":"ContainerStarted","Data":"b37c6ca857d91f383105d36c188d5ba6ee43be117275b57018c908daf13b72b3"} Sep 06 06:13:00.775126 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:00.774667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" event={"ID":"2caba584-fd59-4567-aad7-3287cfd87f21","Type":"ContainerStarted","Data":"1435a620eab08edac8babedcb04fb8016633fbae70b26e7a670e019dd79d1382"} Sep 06 06:13:00.775126 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:00.774783 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" Sep 06 06:13:00.776273 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:00.776247 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" event={"ID":"49aff0de-2f7e-41ec-aadd-d7f38f18a742","Type":"ContainerStarted","Data":"dec7e4a03a12fdb11f34e01ded5cd79dd2db3b43e818578e4223ccc9a6f8130a"} Sep 06 06:13:00.776433 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:00.776289 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" event={"ID":"49aff0de-2f7e-41ec-aadd-d7f38f18a742","Type":"ContainerStarted","Data":"0d6ef047851eba69e6735b31fec8502cc873449920c09df377ff7c5bb129185d"} Sep 06 06:13:00.776433 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:00.776327 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" Sep 06 06:13:00.792706 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:00.792649 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" podStartSLOduration=1.792630091 podStartE2EDuration="1.792630091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:13:00.789948217 +0000 UTC m=+1146.438567419" watchObservedRunningTime="2026-09-06 06:13:00.792630091 +0000 UTC m=+1146.441249274" Sep 06 06:13:00.804818 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:00.804768 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" podStartSLOduration=1.804749841 podStartE2EDuration="1.804749841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:13:00.8044081 +0000 UTC m=+1146.453027281" watchObservedRunningTime="2026-09-06 06:13:00.804749841 +0000 UTC m=+1146.453369023" Sep 06 06:13:11.783723 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:11.783688 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" Sep 06 06:13:11.784205 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:11.783892 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" Sep 06 06:13:12.090511 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.090414 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v"] Sep 06 06:13:12.094143 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.094120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.096779 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.096739 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ceca3b\"/\"kube-root-ca.crt\"" Sep 06 06:13:12.096779 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.096758 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ceca3b\"/\"openshift-service-ca.crt\"" Sep 06 06:13:12.096779 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.096758 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ceca3b\"/\"default-dockercfg-gwjbv\"" Sep 06 06:13:12.097977 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.097948 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ceca3b\"/\"test-model-ceca3b-kserve-self-signed-certs\"" Sep 06 06:13:12.104212 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.104190 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v"] Sep 06 06:13:12.227605 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.227563 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-tmp-dir\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.227791 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.227720 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709f0807-b7ad-489d-82ad-6591f16dad8f-tls-certs\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.227791 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.227763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-model-cache\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.227912 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.227798 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdvgq\" (UniqueName: \"kubernetes.io/projected/709f0807-b7ad-489d-82ad-6591f16dad8f-kube-api-access-wdvgq\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.227912 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.227824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-home\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.227912 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.227889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-dshm\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.328789 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.328741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709f0807-b7ad-489d-82ad-6591f16dad8f-tls-certs\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.328789 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.328796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-model-cache\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.329040 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.328853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdvgq\" (UniqueName: \"kubernetes.io/projected/709f0807-b7ad-489d-82ad-6591f16dad8f-kube-api-access-wdvgq\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.329040 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.328887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-home\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.329040 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.328920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-dshm\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.329040 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.328948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-tmp-dir\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.329357 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.329327 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-model-cache\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.329476 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.329362 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-tmp-dir\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.329476 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.329421 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-home\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.331240 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.331213 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-dshm\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.331487 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.331470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709f0807-b7ad-489d-82ad-6591f16dad8f-tls-certs\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.337329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.337308 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdvgq\" (UniqueName: \"kubernetes.io/projected/709f0807-b7ad-489d-82ad-6591f16dad8f-kube-api-access-wdvgq\") pod \"test-model-ceca3b-kserve-6cf495fbb7-jt84v\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.405158 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.405065 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:12.675658 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.675630 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v"] Sep 06 06:13:12.678428 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:13:12.678395 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod709f0807_b7ad_489d_82ad_6591f16dad8f.slice/crio-871332a6969bd4177d8abc4be3b1f626b1a90a85bf27ea81e86717c463248a3a WatchSource:0}: Error finding container 871332a6969bd4177d8abc4be3b1f626b1a90a85bf27ea81e86717c463248a3a: Status 404 returned error can't find the container with id 871332a6969bd4177d8abc4be3b1f626b1a90a85bf27ea81e86717c463248a3a Sep 06 06:13:12.843837 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.843784 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" event={"ID":"709f0807-b7ad-489d-82ad-6591f16dad8f","Type":"ContainerStarted","Data":"fb6f69913759397d373908532e4b2cb9db74232bb2d1806721598aad394c8070"} Sep 06 06:13:12.844195 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.843845 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" event={"ID":"709f0807-b7ad-489d-82ad-6591f16dad8f","Type":"ContainerStarted","Data":"871332a6969bd4177d8abc4be3b1f626b1a90a85bf27ea81e86717c463248a3a"} Sep 06 06:13:12.860512 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:12.860459 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" podStartSLOduration=0.860439249 podStartE2EDuration="860.439249ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:13:12.859680459 +0000 UTC m=+1158.508299645" watchObservedRunningTime="2026-09-06 06:13:12.860439249 +0000 UTC m=+1158.509058495" Sep 06 06:13:16.643128 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:16.643081 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz"] Sep 06 06:13:22.406150 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:22.406063 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:22.406531 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:22.406330 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:22.419174 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:22.419143 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:22.912111 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:22.912079 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:13:28.510346 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.510308 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg"] Sep 06 06:13:28.511167 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.511112 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" podUID="de73a3c3-016c-4124-82cc-c035ef46c60f" containerName="payload-pre-processing" containerID="cri-o://04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a" gracePeriod=30 Sep 06 06:13:28.776565 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.776500 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" Sep 06 06:13:28.789447 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.789415 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de73a3c3-016c-4124-82cc-c035ef46c60f-plugins-config-volume\") pod \"de73a3c3-016c-4124-82cc-c035ef46c60f\" (UID: \"de73a3c3-016c-4124-82cc-c035ef46c60f\") " Sep 06 06:13:28.789609 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.789506 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th4nh\" (UniqueName: \"kubernetes.io/projected/de73a3c3-016c-4124-82cc-c035ef46c60f-kube-api-access-th4nh\") pod \"de73a3c3-016c-4124-82cc-c035ef46c60f\" (UID: \"de73a3c3-016c-4124-82cc-c035ef46c60f\") " Sep 06 06:13:28.789804 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.789779 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de73a3c3-016c-4124-82cc-c035ef46c60f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de73a3c3-016c-4124-82cc-c035ef46c60f" (UID: "de73a3c3-016c-4124-82cc-c035ef46c60f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:13:28.789946 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.789927 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de73a3c3-016c-4124-82cc-c035ef46c60f-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:13:28.791910 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.791867 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de73a3c3-016c-4124-82cc-c035ef46c60f-kube-api-access-th4nh" (OuterVolumeSpecName: "kube-api-access-th4nh") pod "de73a3c3-016c-4124-82cc-c035ef46c60f" (UID: "de73a3c3-016c-4124-82cc-c035ef46c60f"). InnerVolumeSpecName "kube-api-access-th4nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:13:28.890696 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.890670 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th4nh\" (UniqueName: \"kubernetes.io/projected/de73a3c3-016c-4124-82cc-c035ef46c60f-kube-api-access-th4nh\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:13:28.932775 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.932743 2580 generic.go:356] "Generic (PLEG): container finished" podID="de73a3c3-016c-4124-82cc-c035ef46c60f" containerID="04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a" exitCode=0 Sep 06 06:13:28.932988 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.932787 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" Sep 06 06:13:28.932988 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.932869 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" event={"ID":"de73a3c3-016c-4124-82cc-c035ef46c60f","Type":"ContainerDied","Data":"04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a"} Sep 06 06:13:28.932988 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.932928 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg" event={"ID":"de73a3c3-016c-4124-82cc-c035ef46c60f","Type":"ContainerDied","Data":"2bd5139c9fd15b586de84178552cd636c9ba4b89b50c45f3392c65f238b4ae74"} Sep 06 06:13:28.932988 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.932961 2580 scope.go:117] "RemoveContainer" containerID="04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a" Sep 06 06:13:28.941883 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.941865 2580 scope.go:117] "RemoveContainer" containerID="04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a" Sep 06 06:13:28.942178 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:13:28.942159 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a\": container with ID starting with 04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a not found: ID does not exist" containerID="04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a" Sep 06 06:13:28.942239 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.942187 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a"} err="failed to get container status \"04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a\": rpc error: code = NotFound desc = could not find container \"04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a\": container with ID starting with 04dad2388e623f35082c11985da7599314c6f5daa52217d52b3a9002e116e68a not found: ID does not exist" Sep 06 06:13:28.955194 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.953975 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg"] Sep 06 06:13:28.960866 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:28.960840 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e75768-65f696b578-2m9tg"] Sep 06 06:13:29.702538 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.702505 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk"] Sep 06 06:13:29.702986 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.702736 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" podUID="49aff0de-2f7e-41ec-aadd-d7f38f18a742" containerName="payload-processing" containerID="cri-o://dec7e4a03a12fdb11f34e01ded5cd79dd2db3b43e818578e4223ccc9a6f8130a" gracePeriod=30 Sep 06 06:13:29.710205 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.710174 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr"] Sep 06 06:13:29.710938 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.710908 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" podUID="2caba584-fd59-4567-aad7-3287cfd87f21" containerName="payload-pre-processing" containerID="cri-o://b37c6ca857d91f383105d36c188d5ba6ee43be117275b57018c908daf13b72b3" gracePeriod=30 Sep 06 06:13:29.942261 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.942223 2580 generic.go:356] "Generic (PLEG): container finished" podID="2caba584-fd59-4567-aad7-3287cfd87f21" containerID="b37c6ca857d91f383105d36c188d5ba6ee43be117275b57018c908daf13b72b3" exitCode=0 Sep 06 06:13:29.942444 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.942287 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" event={"ID":"2caba584-fd59-4567-aad7-3287cfd87f21","Type":"ContainerDied","Data":"b37c6ca857d91f383105d36c188d5ba6ee43be117275b57018c908daf13b72b3"} Sep 06 06:13:29.945284 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.945260 2580 generic.go:356] "Generic (PLEG): container finished" podID="49aff0de-2f7e-41ec-aadd-d7f38f18a742" containerID="dec7e4a03a12fdb11f34e01ded5cd79dd2db3b43e818578e4223ccc9a6f8130a" exitCode=0 Sep 06 06:13:29.945418 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.945299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" event={"ID":"49aff0de-2f7e-41ec-aadd-d7f38f18a742","Type":"ContainerDied","Data":"dec7e4a03a12fdb11f34e01ded5cd79dd2db3b43e818578e4223ccc9a6f8130a"} Sep 06 06:13:29.990227 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.990194 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" Sep 06 06:13:29.994311 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.994281 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" Sep 06 06:13:29.999506 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.999480 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caba584-fd59-4567-aad7-3287cfd87f21-plugins-config-volume\") pod \"2caba584-fd59-4567-aad7-3287cfd87f21\" (UID: \"2caba584-fd59-4567-aad7-3287cfd87f21\") " Sep 06 06:13:29.999659 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.999514 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-872jl\" (UniqueName: \"kubernetes.io/projected/2caba584-fd59-4567-aad7-3287cfd87f21-kube-api-access-872jl\") pod \"2caba584-fd59-4567-aad7-3287cfd87f21\" (UID: \"2caba584-fd59-4567-aad7-3287cfd87f21\") " Sep 06 06:13:29.999659 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.999631 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49aff0de-2f7e-41ec-aadd-d7f38f18a742-plugins-config-volume\") pod \"49aff0de-2f7e-41ec-aadd-d7f38f18a742\" (UID: \"49aff0de-2f7e-41ec-aadd-d7f38f18a742\") " Sep 06 06:13:29.999770 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.999685 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vvcg\" (UniqueName: \"kubernetes.io/projected/49aff0de-2f7e-41ec-aadd-d7f38f18a742-kube-api-access-5vvcg\") pod \"49aff0de-2f7e-41ec-aadd-d7f38f18a742\" (UID: \"49aff0de-2f7e-41ec-aadd-d7f38f18a742\") " Sep 06 06:13:30.000229 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:29.999928 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2caba584-fd59-4567-aad7-3287cfd87f21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2caba584-fd59-4567-aad7-3287cfd87f21" (UID: "2caba584-fd59-4567-aad7-3287cfd87f21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:13:30.000229 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.000032 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caba584-fd59-4567-aad7-3287cfd87f21-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:13:30.000229 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.000046 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49aff0de-2f7e-41ec-aadd-d7f38f18a742-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49aff0de-2f7e-41ec-aadd-d7f38f18a742" (UID: "49aff0de-2f7e-41ec-aadd-d7f38f18a742"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:13:30.003580 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.003552 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2caba584-fd59-4567-aad7-3287cfd87f21-kube-api-access-872jl" (OuterVolumeSpecName: "kube-api-access-872jl") pod "2caba584-fd59-4567-aad7-3287cfd87f21" (UID: "2caba584-fd59-4567-aad7-3287cfd87f21"). InnerVolumeSpecName "kube-api-access-872jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:13:30.005005 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.004982 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49aff0de-2f7e-41ec-aadd-d7f38f18a742-kube-api-access-5vvcg" (OuterVolumeSpecName: "kube-api-access-5vvcg") pod "49aff0de-2f7e-41ec-aadd-d7f38f18a742" (UID: "49aff0de-2f7e-41ec-aadd-d7f38f18a742"). InnerVolumeSpecName "kube-api-access-5vvcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:13:30.100764 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.100724 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49aff0de-2f7e-41ec-aadd-d7f38f18a742-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:13:30.100764 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.100757 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vvcg\" (UniqueName: \"kubernetes.io/projected/49aff0de-2f7e-41ec-aadd-d7f38f18a742-kube-api-access-5vvcg\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:13:30.100764 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.100767 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-872jl\" (UniqueName: \"kubernetes.io/projected/2caba584-fd59-4567-aad7-3287cfd87f21-kube-api-access-872jl\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:13:30.875483 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.875449 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de73a3c3-016c-4124-82cc-c035ef46c60f" path="/var/lib/kubelet/pods/de73a3c3-016c-4124-82cc-c035ef46c60f/volumes" Sep 06 06:13:30.952205 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.952181 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" Sep 06 06:13:30.952382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.952219 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr" event={"ID":"2caba584-fd59-4567-aad7-3287cfd87f21","Type":"ContainerDied","Data":"1435a620eab08edac8babedcb04fb8016633fbae70b26e7a670e019dd79d1382"} Sep 06 06:13:30.952382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.952289 2580 scope.go:117] "RemoveContainer" containerID="b37c6ca857d91f383105d36c188d5ba6ee43be117275b57018c908daf13b72b3" Sep 06 06:13:30.953621 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.953517 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" event={"ID":"49aff0de-2f7e-41ec-aadd-d7f38f18a742","Type":"ContainerDied","Data":"0d6ef047851eba69e6735b31fec8502cc873449920c09df377ff7c5bb129185d"} Sep 06 06:13:30.953621 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.953561 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk" Sep 06 06:13:30.962876 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.962858 2580 scope.go:117] "RemoveContainer" containerID="dec7e4a03a12fdb11f34e01ded5cd79dd2db3b43e818578e4223ccc9a6f8130a" Sep 06 06:13:30.979768 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.979746 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr"] Sep 06 06:13:30.983353 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.983329 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f260fc83-6cbbcdd9fd-gsbvr"] Sep 06 06:13:30.994133 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.994105 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk"] Sep 06 06:13:30.998476 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:30.998454 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f260fc83-5c597f9665-5lkxk"] Sep 06 06:13:31.669888 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:31.669849 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" podUID="1cce4976-e9bf-4d16-98a1-c574323ff5e2" containerName="main" containerID="cri-o://1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6" gracePeriod=45 Sep 06 06:13:31.926729 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:31.926670 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:13:31.961166 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:31.961140 2580 generic.go:356] "Generic (PLEG): container finished" podID="1cce4976-e9bf-4d16-98a1-c574323ff5e2" containerID="1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6" exitCode=0 Sep 06 06:13:31.961316 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:31.961182 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" Sep 06 06:13:31.961316 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:31.961228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" event={"ID":"1cce4976-e9bf-4d16-98a1-c574323ff5e2","Type":"ContainerDied","Data":"1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6"} Sep 06 06:13:31.961316 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:31.961271 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz" event={"ID":"1cce4976-e9bf-4d16-98a1-c574323ff5e2","Type":"ContainerDied","Data":"fe496a59b45ef0be87866b2e1405d0c06fd633efdbc2f232c71f86ebdd639bd4"} Sep 06 06:13:31.961316 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:31.961294 2580 scope.go:117] "RemoveContainer" containerID="1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6" Sep 06 06:13:31.969739 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:31.969719 2580 scope.go:117] "RemoveContainer" containerID="1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6" Sep 06 06:13:31.970042 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:13:31.970021 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6\": container with ID starting with 1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6 not found: ID does not exist" containerID="1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6" Sep 06 06:13:31.970106 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:31.970050 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6"} err="failed to get container status \"1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6\": rpc error: code = NotFound desc = could not find container \"1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6\": container with ID starting with 1591d10b914080d0c7c7d48e08a55a4b3ae125830a3eee8c5d444966708a9ba6 not found: ID does not exist" Sep 06 06:13:32.018019 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.017996 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-home\") pod \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " Sep 06 06:13:32.018116 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.018050 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfnr6\" (UniqueName: \"kubernetes.io/projected/1cce4976-e9bf-4d16-98a1-c574323ff5e2-kube-api-access-wfnr6\") pod \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " Sep 06 06:13:32.018116 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.018092 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-dshm\") pod \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " Sep 06 06:13:32.018213 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.018167 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-tmp-dir\") pod \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " Sep 06 06:13:32.018213 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.018209 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cce4976-e9bf-4d16-98a1-c574323ff5e2-tls-certs\") pod \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " Sep 06 06:13:32.018326 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.018262 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-model-cache\") pod \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\" (UID: \"1cce4976-e9bf-4d16-98a1-c574323ff5e2\") " Sep 06 06:13:32.018326 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.018289 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-home" (OuterVolumeSpecName: "home") pod "1cce4976-e9bf-4d16-98a1-c574323ff5e2" (UID: "1cce4976-e9bf-4d16-98a1-c574323ff5e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:32.018430 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.018409 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1cce4976-e9bf-4d16-98a1-c574323ff5e2" (UID: "1cce4976-e9bf-4d16-98a1-c574323ff5e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:32.018544 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.018524 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "1cce4976-e9bf-4d16-98a1-c574323ff5e2" (UID: "1cce4976-e9bf-4d16-98a1-c574323ff5e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:32.018627 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.018610 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-tmp-dir\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:13:32.018705 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.018632 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-model-cache\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:13:32.018705 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.018645 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-home\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:13:32.020155 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.020133 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cce4976-e9bf-4d16-98a1-c574323ff5e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cce4976-e9bf-4d16-98a1-c574323ff5e2" (UID: "1cce4976-e9bf-4d16-98a1-c574323ff5e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:13:32.020231 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.020190 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-dshm" (OuterVolumeSpecName: "dshm") pod "1cce4976-e9bf-4d16-98a1-c574323ff5e2" (UID: "1cce4976-e9bf-4d16-98a1-c574323ff5e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:32.020231 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.020205 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cce4976-e9bf-4d16-98a1-c574323ff5e2-kube-api-access-wfnr6" (OuterVolumeSpecName: "kube-api-access-wfnr6") pod "1cce4976-e9bf-4d16-98a1-c574323ff5e2" (UID: "1cce4976-e9bf-4d16-98a1-c574323ff5e2"). InnerVolumeSpecName "kube-api-access-wfnr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:13:32.119893 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.119865 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cce4976-e9bf-4d16-98a1-c574323ff5e2-tls-certs\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:13:32.119893 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.119887 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfnr6\" (UniqueName: \"kubernetes.io/projected/1cce4976-e9bf-4d16-98a1-c574323ff5e2-kube-api-access-wfnr6\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:13:32.119893 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.119896 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cce4976-e9bf-4d16-98a1-c574323ff5e2-dshm\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:13:32.286257 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.286224 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz"] Sep 06 06:13:32.289981 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.289956 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-d5939f9b-kserve-d9877447d-8vccz"] Sep 06 06:13:32.876029 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.875998 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cce4976-e9bf-4d16-98a1-c574323ff5e2" path="/var/lib/kubelet/pods/1cce4976-e9bf-4d16-98a1-c574323ff5e2/volumes" Sep 06 06:13:32.876375 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.876362 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2caba584-fd59-4567-aad7-3287cfd87f21" path="/var/lib/kubelet/pods/2caba584-fd59-4567-aad7-3287cfd87f21/volumes" Sep 06 06:13:32.876672 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:32.876660 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49aff0de-2f7e-41ec-aadd-d7f38f18a742" path="/var/lib/kubelet/pods/49aff0de-2f7e-41ec-aadd-d7f38f18a742/volumes" Sep 06 06:13:40.236862 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.236814 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs"] Sep 06 06:13:40.237345 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.237329 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de73a3c3-016c-4124-82cc-c035ef46c60f" containerName="payload-pre-processing" Sep 06 06:13:40.237390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.237347 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="de73a3c3-016c-4124-82cc-c035ef46c60f" containerName="payload-pre-processing" Sep 06 06:13:40.237390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.237361 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2caba584-fd59-4567-aad7-3287cfd87f21" containerName="payload-pre-processing" Sep 06 06:13:40.237390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.237367 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2caba584-fd59-4567-aad7-3287cfd87f21" containerName="payload-pre-processing" Sep 06 06:13:40.237390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.237375 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cce4976-e9bf-4d16-98a1-c574323ff5e2" containerName="main" Sep 06 06:13:40.237390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.237382 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cce4976-e9bf-4d16-98a1-c574323ff5e2" containerName="main" Sep 06 06:13:40.237583 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.237398 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49aff0de-2f7e-41ec-aadd-d7f38f18a742" containerName="payload-processing" Sep 06 06:13:40.237583 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.237404 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="49aff0de-2f7e-41ec-aadd-d7f38f18a742" containerName="payload-processing" Sep 06 06:13:40.237583 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.237471 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="de73a3c3-016c-4124-82cc-c035ef46c60f" containerName="payload-pre-processing" Sep 06 06:13:40.237583 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.237480 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="49aff0de-2f7e-41ec-aadd-d7f38f18a742" containerName="payload-processing" Sep 06 06:13:40.237583 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.237489 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2caba584-fd59-4567-aad7-3287cfd87f21" containerName="payload-pre-processing" Sep 06 06:13:40.237583 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.237497 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cce4976-e9bf-4d16-98a1-c574323ff5e2" containerName="main" Sep 06 06:13:40.241690 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.241674 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" Sep 06 06:13:40.244682 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.244651 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c50b6f\"" Sep 06 06:13:40.244893 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.244703 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c50b6f-dockercfg-zmbxg\"" Sep 06 06:13:40.251396 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.251369 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs"] Sep 06 06:13:40.300646 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.300613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5n58\" (UniqueName: \"kubernetes.io/projected/395fc0c4-3e26-44e6-bfb4-98b9fd04632a-kube-api-access-m5n58\") pod \"payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs\" (UID: \"395fc0c4-3e26-44e6-bfb4-98b9fd04632a\") " pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" Sep 06 06:13:40.300819 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.300695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395fc0c4-3e26-44e6-bfb4-98b9fd04632a-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs\" (UID: \"395fc0c4-3e26-44e6-bfb4-98b9fd04632a\") " pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" Sep 06 06:13:40.401238 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.401203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5n58\" (UniqueName: \"kubernetes.io/projected/395fc0c4-3e26-44e6-bfb4-98b9fd04632a-kube-api-access-m5n58\") pod \"payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs\" (UID: \"395fc0c4-3e26-44e6-bfb4-98b9fd04632a\") " pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" Sep 06 06:13:40.401392 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.401253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395fc0c4-3e26-44e6-bfb4-98b9fd04632a-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs\" (UID: \"395fc0c4-3e26-44e6-bfb4-98b9fd04632a\") " pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" Sep 06 06:13:40.401973 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.401953 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395fc0c4-3e26-44e6-bfb4-98b9fd04632a-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs\" (UID: \"395fc0c4-3e26-44e6-bfb4-98b9fd04632a\") " pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" Sep 06 06:13:40.410843 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.410806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5n58\" (UniqueName: \"kubernetes.io/projected/395fc0c4-3e26-44e6-bfb4-98b9fd04632a-kube-api-access-m5n58\") pod \"payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs\" (UID: \"395fc0c4-3e26-44e6-bfb4-98b9fd04632a\") " pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" Sep 06 06:13:40.553056 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.552975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" Sep 06 06:13:40.826088 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:40.826060 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs"] Sep 06 06:13:40.828743 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:13:40.828715 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395fc0c4_3e26_44e6_bfb4_98b9fd04632a.slice/crio-e119df7d399fb63a8801196ddc484510763d47516bd09b301870f951425deaa2 WatchSource:0}: Error finding container e119df7d399fb63a8801196ddc484510763d47516bd09b301870f951425deaa2: Status 404 returned error can't find the container with id e119df7d399fb63a8801196ddc484510763d47516bd09b301870f951425deaa2 Sep 06 06:13:41.010937 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:41.010905 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" event={"ID":"395fc0c4-3e26-44e6-bfb4-98b9fd04632a","Type":"ContainerStarted","Data":"67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada"} Sep 06 06:13:41.011113 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:41.010959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" event={"ID":"395fc0c4-3e26-44e6-bfb4-98b9fd04632a","Type":"ContainerStarted","Data":"e119df7d399fb63a8801196ddc484510763d47516bd09b301870f951425deaa2"} Sep 06 06:13:41.011113 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:41.011008 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" Sep 06 06:13:41.028007 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:41.027954 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" podStartSLOduration=1.027937054 podStartE2EDuration="1.027937054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:13:41.026105945 +0000 UTC m=+1186.674725146" watchObservedRunningTime="2026-09-06 06:13:41.027937054 +0000 UTC m=+1186.676556232" Sep 06 06:13:52.019424 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:52.019397 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" Sep 06 06:13:58.030027 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.029990 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw"] Sep 06 06:13:58.033434 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.033416 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.036083 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.036016 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0bce0b\"/\"kube-root-ca.crt\"" Sep 06 06:13:58.036304 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.036288 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0bce0b\"/\"openshift-service-ca.crt\"" Sep 06 06:13:58.037473 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.037451 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0bce0b\"/\"default-dockercfg-p6q27\"" Sep 06 06:13:58.037552 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.037491 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0bce0b\"/\"test-model-0bce0b-kserve-self-signed-certs\"" Sep 06 06:13:58.043840 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.043800 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw"] Sep 06 06:13:58.173308 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.173272 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-dshm\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.173480 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.173326 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-home\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.173480 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.173381 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-tmp-dir\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.173480 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.173437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5cdx\" (UniqueName: \"kubernetes.io/projected/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-kube-api-access-p5cdx\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.173480 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.173461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-model-cache\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.173622 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.173542 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-tls-certs\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.274097 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.274058 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-home\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.274285 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.274105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-tmp-dir\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.274285 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.274151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5cdx\" (UniqueName: \"kubernetes.io/projected/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-kube-api-access-p5cdx\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.274285 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.274175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-model-cache\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.274285 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.274214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-tls-certs\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.274514 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.274298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-dshm\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.274514 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.274489 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-home\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.274630 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.274521 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-tmp-dir\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.274630 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.274557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-model-cache\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.276633 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.276602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-dshm\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.276746 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.276733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-tls-certs\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.281647 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.281594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5cdx\" (UniqueName: \"kubernetes.io/projected/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-kube-api-access-p5cdx\") pod \"test-model-0bce0b-kserve-666dbb4869-rfntw\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.344291 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.344255 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:13:58.613607 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:58.613571 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw"] Sep 06 06:13:58.614614 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:13:58.614582 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2746cc5f_1b29_4182_a1b0_3c0ae93737e6.slice/crio-3c4624856503d70cc73cf3bd785feece683f9d470ffc7c71ac1ad7dae847f30f WatchSource:0}: Error finding container 3c4624856503d70cc73cf3bd785feece683f9d470ffc7c71ac1ad7dae847f30f: Status 404 returned error can't find the container with id 3c4624856503d70cc73cf3bd785feece683f9d470ffc7c71ac1ad7dae847f30f Sep 06 06:13:59.108710 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:59.108672 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" event={"ID":"2746cc5f-1b29-4182-a1b0-3c0ae93737e6","Type":"ContainerStarted","Data":"5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918"} Sep 06 06:13:59.109136 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:59.108721 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" event={"ID":"2746cc5f-1b29-4182-a1b0-3c0ae93737e6","Type":"ContainerStarted","Data":"3c4624856503d70cc73cf3bd785feece683f9d470ffc7c71ac1ad7dae847f30f"} Sep 06 06:13:59.125708 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:13:59.125662 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" podStartSLOduration=1.125649002 podStartE2EDuration="1.125649002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:13:59.124725361 +0000 UTC m=+1204.773344543" watchObservedRunningTime="2026-09-06 06:13:59.125649002 +0000 UTC m=+1204.774268181" Sep 06 06:14:08.344481 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:08.344433 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:14:08.344481 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:08.344499 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:14:08.358780 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:08.358751 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:14:09.173583 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:09.173552 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:14:37.785820 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:37.785784 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs"] Sep 06 06:14:37.786734 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:37.786678 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" podUID="395fc0c4-3e26-44e6-bfb4-98b9fd04632a" containerName="payload-processing" containerID="cri-o://67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada" gracePeriod=30 Sep 06 06:14:38.035555 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.035528 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" Sep 06 06:14:38.160089 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.159993 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5n58\" (UniqueName: \"kubernetes.io/projected/395fc0c4-3e26-44e6-bfb4-98b9fd04632a-kube-api-access-m5n58\") pod \"395fc0c4-3e26-44e6-bfb4-98b9fd04632a\" (UID: \"395fc0c4-3e26-44e6-bfb4-98b9fd04632a\") " Sep 06 06:14:38.160283 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.160095 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395fc0c4-3e26-44e6-bfb4-98b9fd04632a-plugins-config-volume\") pod \"395fc0c4-3e26-44e6-bfb4-98b9fd04632a\" (UID: \"395fc0c4-3e26-44e6-bfb4-98b9fd04632a\") " Sep 06 06:14:38.160491 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.160465 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395fc0c4-3e26-44e6-bfb4-98b9fd04632a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "395fc0c4-3e26-44e6-bfb4-98b9fd04632a" (UID: "395fc0c4-3e26-44e6-bfb4-98b9fd04632a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:14:38.162224 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.162199 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395fc0c4-3e26-44e6-bfb4-98b9fd04632a-kube-api-access-m5n58" (OuterVolumeSpecName: "kube-api-access-m5n58") pod "395fc0c4-3e26-44e6-bfb4-98b9fd04632a" (UID: "395fc0c4-3e26-44e6-bfb4-98b9fd04632a"). InnerVolumeSpecName "kube-api-access-m5n58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:14:38.261653 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.261615 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5n58\" (UniqueName: \"kubernetes.io/projected/395fc0c4-3e26-44e6-bfb4-98b9fd04632a-kube-api-access-m5n58\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:14:38.261653 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.261647 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395fc0c4-3e26-44e6-bfb4-98b9fd04632a-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:14:38.311400 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.311368 2580 generic.go:356] "Generic (PLEG): container finished" podID="395fc0c4-3e26-44e6-bfb4-98b9fd04632a" containerID="67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada" exitCode=0 Sep 06 06:14:38.311571 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.311412 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" Sep 06 06:14:38.311571 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.311434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" event={"ID":"395fc0c4-3e26-44e6-bfb4-98b9fd04632a","Type":"ContainerDied","Data":"67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada"} Sep 06 06:14:38.311571 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.311475 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs" event={"ID":"395fc0c4-3e26-44e6-bfb4-98b9fd04632a","Type":"ContainerDied","Data":"e119df7d399fb63a8801196ddc484510763d47516bd09b301870f951425deaa2"} Sep 06 06:14:38.311571 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.311497 2580 scope.go:117] "RemoveContainer" containerID="67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada" Sep 06 06:14:38.321055 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.321036 2580 scope.go:117] "RemoveContainer" containerID="67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada" Sep 06 06:14:38.321332 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:14:38.321311 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada\": container with ID starting with 67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada not found: ID does not exist" containerID="67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada" Sep 06 06:14:38.321391 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.321345 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada"} err="failed to get container status \"67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada\": rpc error: code = NotFound desc = could not find container \"67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada\": container with ID starting with 67e23da481bde6312adeb4e5554bec9689168ad532b8a25c2dc38dbb8758aada not found: ID does not exist" Sep 06 06:14:38.332534 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.332507 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs"] Sep 06 06:14:38.336186 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.336165 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c50b6f-848bdf7b85-bx9fs"] Sep 06 06:14:38.878347 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:14:38.878311 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="395fc0c4-3e26-44e6-bfb4-98b9fd04632a" path="/var/lib/kubelet/pods/395fc0c4-3e26-44e6-bfb4-98b9fd04632a/volumes" Sep 06 06:15:00.136177 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:00.136138 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj"] Sep 06 06:15:00.136611 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:00.136561 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="395fc0c4-3e26-44e6-bfb4-98b9fd04632a" containerName="payload-processing" Sep 06 06:15:00.136611 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:00.136574 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="395fc0c4-3e26-44e6-bfb4-98b9fd04632a" containerName="payload-processing" Sep 06 06:15:00.136683 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:00.136656 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="395fc0c4-3e26-44e6-bfb4-98b9fd04632a" containerName="payload-processing" Sep 06 06:15:00.140768 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:00.140751 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj" Sep 06 06:15:00.143149 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:00.143127 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:15:00.147423 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:00.147403 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj"] Sep 06 06:15:00.151597 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:00.151578 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj" Sep 06 06:15:00.436289 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:00.436236 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj"] Sep 06 06:15:00.439392 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:15:00.439355 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138423f6_da26_4506_956f_4033318f776a.slice/crio-cb57e9e03a0d4959bd576e59bc6efbce1311c4a13dc01ffdad51573515fd695c WatchSource:0}: Error finding container cb57e9e03a0d4959bd576e59bc6efbce1311c4a13dc01ffdad51573515fd695c: Status 404 returned error can't find the container with id cb57e9e03a0d4959bd576e59bc6efbce1311c4a13dc01ffdad51573515fd695c Sep 06 06:15:00.441274 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:00.441257 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 06:15:01.436225 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:01.436185 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj" event={"ID":"138423f6-da26-4506-956f-4033318f776a","Type":"ContainerStarted","Data":"cb57e9e03a0d4959bd576e59bc6efbce1311c4a13dc01ffdad51573515fd695c"} Sep 06 06:15:03.448355 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:03.448268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj" event={"ID":"138423f6-da26-4506-956f-4033318f776a","Type":"ContainerStarted","Data":"baff5a6b5b33bfaf81fb98a6b1564304d41d5c2ce9a5029aa9201655bdc750d6"} Sep 06 06:15:03.464670 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:03.464620 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj" podStartSLOduration=1.017628664 podStartE2EDuration="3.464605868s" totalImagesPullingTime="2.446977204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:15:03.462180989 +0000 UTC m=+1269.110800167" watchObservedRunningTime="2026-09-06 06:15:03.464605868 +0000 UTC m=+1269.113225046" Sep 06 06:15:04.455869 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:04.455813 2580 generic.go:356] "Generic (PLEG): container finished" podID="138423f6-da26-4506-956f-4033318f776a" containerID="baff5a6b5b33bfaf81fb98a6b1564304d41d5c2ce9a5029aa9201655bdc750d6" exitCode=0 Sep 06 06:15:04.456296 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:04.455873 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj" event={"ID":"138423f6-da26-4506-956f-4033318f776a","Type":"ContainerDied","Data":"baff5a6b5b33bfaf81fb98a6b1564304d41d5c2ce9a5029aa9201655bdc750d6"} Sep 06 06:15:05.583676 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:05.583651 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj" Sep 06 06:15:06.467181 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:06.467152 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj" Sep 06 06:15:06.467348 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:06.467184 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj" event={"ID":"138423f6-da26-4506-956f-4033318f776a","Type":"ContainerDied","Data":"cb57e9e03a0d4959bd576e59bc6efbce1311c4a13dc01ffdad51573515fd695c"} Sep 06 06:15:06.467348 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:06.467226 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb57e9e03a0d4959bd576e59bc6efbce1311c4a13dc01ffdad51573515fd695c" Sep 06 06:15:08.812359 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:08.812324 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt"] Sep 06 06:15:08.812951 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:08.812930 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="138423f6-da26-4506-956f-4033318f776a" containerName="cleanup" Sep 06 06:15:08.812951 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:08.812952 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="138423f6-da26-4506-956f-4033318f776a" containerName="cleanup" Sep 06 06:15:08.813105 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:08.813089 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="138423f6-da26-4506-956f-4033318f776a" containerName="cleanup" Sep 06 06:15:08.816172 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:08.816149 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" Sep 06 06:15:08.818657 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:08.818635 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fc95779d-dockercfg-66dx6\"" Sep 06 06:15:08.819000 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:08.818975 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fc95779d\"" Sep 06 06:15:08.827017 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:08.826992 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt"] Sep 06 06:15:08.952558 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:08.952524 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6psfn\" (UniqueName: \"kubernetes.io/projected/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8-kube-api-access-6psfn\") pod \"payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt\" (UID: \"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8\") " pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" Sep 06 06:15:08.952751 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:08.952640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt\" (UID: \"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8\") " pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" Sep 06 06:15:09.053561 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:09.053529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt\" (UID: \"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8\") " pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" Sep 06 06:15:09.053736 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:09.053689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6psfn\" (UniqueName: \"kubernetes.io/projected/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8-kube-api-access-6psfn\") pod \"payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt\" (UID: \"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8\") " pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" Sep 06 06:15:09.054283 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:09.054258 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt\" (UID: \"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8\") " pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" Sep 06 06:15:09.061478 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:09.061455 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6psfn\" (UniqueName: \"kubernetes.io/projected/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8-kube-api-access-6psfn\") pod \"payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt\" (UID: \"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8\") " pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" Sep 06 06:15:09.128690 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:09.128618 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" Sep 06 06:15:09.398703 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:09.398553 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt"] Sep 06 06:15:09.401211 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:15:09.401186 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de7ae89_45d2_44ee_b8b4_e8cf473f09d8.slice/crio-1e30249b4177caa3c57c31e4267aa59f4d7f1b741e59d95647d6e31766d27b56 WatchSource:0}: Error finding container 1e30249b4177caa3c57c31e4267aa59f4d7f1b741e59d95647d6e31766d27b56: Status 404 returned error can't find the container with id 1e30249b4177caa3c57c31e4267aa59f4d7f1b741e59d95647d6e31766d27b56 Sep 06 06:15:09.484569 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:09.484538 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" event={"ID":"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8","Type":"ContainerStarted","Data":"24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7"} Sep 06 06:15:09.484705 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:09.484598 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" event={"ID":"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8","Type":"ContainerStarted","Data":"1e30249b4177caa3c57c31e4267aa59f4d7f1b741e59d95647d6e31766d27b56"} Sep 06 06:15:09.484770 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:09.484749 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" Sep 06 06:15:09.501797 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:09.501750 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" podStartSLOduration=1.501736253 podStartE2EDuration="1.501736253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:15:09.499313036 +0000 UTC m=+1275.147932215" watchObservedRunningTime="2026-09-06 06:15:09.501736253 +0000 UTC m=+1275.150355459" Sep 06 06:15:15.390901 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:15.390854 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v"] Sep 06 06:15:17.794159 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:17.794124 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw"] Sep 06 06:15:20.492181 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:20.492148 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" Sep 06 06:15:30.411446 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.411399 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" podUID="709f0807-b7ad-489d-82ad-6591f16dad8f" containerName="main" containerID="cri-o://fb6f69913759397d373908532e4b2cb9db74232bb2d1806721598aad394c8070" gracePeriod=45 Sep 06 06:15:30.598392 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.598363 2580 generic.go:356] "Generic (PLEG): container finished" podID="709f0807-b7ad-489d-82ad-6591f16dad8f" containerID="fb6f69913759397d373908532e4b2cb9db74232bb2d1806721598aad394c8070" exitCode=0 Sep 06 06:15:30.598530 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.598413 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" event={"ID":"709f0807-b7ad-489d-82ad-6591f16dad8f","Type":"ContainerDied","Data":"fb6f69913759397d373908532e4b2cb9db74232bb2d1806721598aad394c8070"} Sep 06 06:15:30.658169 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.658146 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:15:30.750861 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.750812 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-tmp-dir\") pod \"709f0807-b7ad-489d-82ad-6591f16dad8f\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " Sep 06 06:15:30.751037 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.750869 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-model-cache\") pod \"709f0807-b7ad-489d-82ad-6591f16dad8f\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " Sep 06 06:15:30.751037 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.750932 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdvgq\" (UniqueName: \"kubernetes.io/projected/709f0807-b7ad-489d-82ad-6591f16dad8f-kube-api-access-wdvgq\") pod \"709f0807-b7ad-489d-82ad-6591f16dad8f\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " Sep 06 06:15:30.751037 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.750948 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-dshm\") pod \"709f0807-b7ad-489d-82ad-6591f16dad8f\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " Sep 06 06:15:30.751037 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.750966 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709f0807-b7ad-489d-82ad-6591f16dad8f-tls-certs\") pod \"709f0807-b7ad-489d-82ad-6591f16dad8f\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " Sep 06 06:15:30.751261 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.751065 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-home\") pod \"709f0807-b7ad-489d-82ad-6591f16dad8f\" (UID: \"709f0807-b7ad-489d-82ad-6591f16dad8f\") " Sep 06 06:15:30.751261 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.751068 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "709f0807-b7ad-489d-82ad-6591f16dad8f" (UID: "709f0807-b7ad-489d-82ad-6591f16dad8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:30.751261 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.751192 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-model-cache" (OuterVolumeSpecName: "model-cache") pod "709f0807-b7ad-489d-82ad-6591f16dad8f" (UID: "709f0807-b7ad-489d-82ad-6591f16dad8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:30.751443 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.751346 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-home" (OuterVolumeSpecName: "home") pod "709f0807-b7ad-489d-82ad-6591f16dad8f" (UID: "709f0807-b7ad-489d-82ad-6591f16dad8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:30.751503 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.751464 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-model-cache\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:30.751503 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.751482 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-home\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:30.751503 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.751495 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-tmp-dir\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:30.753178 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.753149 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709f0807-b7ad-489d-82ad-6591f16dad8f-kube-api-access-wdvgq" (OuterVolumeSpecName: "kube-api-access-wdvgq") pod "709f0807-b7ad-489d-82ad-6591f16dad8f" (UID: "709f0807-b7ad-489d-82ad-6591f16dad8f"). InnerVolumeSpecName "kube-api-access-wdvgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:15:30.753271 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.753185 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-dshm" (OuterVolumeSpecName: "dshm") pod "709f0807-b7ad-489d-82ad-6591f16dad8f" (UID: "709f0807-b7ad-489d-82ad-6591f16dad8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:30.753271 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.753188 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709f0807-b7ad-489d-82ad-6591f16dad8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "709f0807-b7ad-489d-82ad-6591f16dad8f" (UID: "709f0807-b7ad-489d-82ad-6591f16dad8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:15:30.852361 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.852333 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdvgq\" (UniqueName: \"kubernetes.io/projected/709f0807-b7ad-489d-82ad-6591f16dad8f-kube-api-access-wdvgq\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:30.852361 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.852358 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709f0807-b7ad-489d-82ad-6591f16dad8f-dshm\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:30.852508 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:30.852368 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709f0807-b7ad-489d-82ad-6591f16dad8f-tls-certs\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:31.604468 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:31.604432 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" Sep 06 06:15:31.604880 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:31.604439 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v" event={"ID":"709f0807-b7ad-489d-82ad-6591f16dad8f","Type":"ContainerDied","Data":"871332a6969bd4177d8abc4be3b1f626b1a90a85bf27ea81e86717c463248a3a"} Sep 06 06:15:31.604880 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:31.604557 2580 scope.go:117] "RemoveContainer" containerID="fb6f69913759397d373908532e4b2cb9db74232bb2d1806721598aad394c8070" Sep 06 06:15:31.623923 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:31.623898 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v"] Sep 06 06:15:31.628314 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:31.628291 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ceca3b/test-model-ceca3b-kserve-6cf495fbb7-jt84v"] Sep 06 06:15:32.814863 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:32.814781 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" podUID="2746cc5f-1b29-4182-a1b0-3c0ae93737e6" containerName="main" containerID="cri-o://5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918" gracePeriod=45 Sep 06 06:15:32.876087 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:32.876060 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="709f0807-b7ad-489d-82ad-6591f16dad8f" path="/var/lib/kubelet/pods/709f0807-b7ad-489d-82ad-6591f16dad8f/volumes" Sep 06 06:15:33.072393 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.072331 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:15:33.173735 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.173707 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-tls-certs\") pod \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " Sep 06 06:15:33.173899 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.173747 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-model-cache\") pod \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " Sep 06 06:15:33.173899 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.173786 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-tmp-dir\") pod \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " Sep 06 06:15:33.173899 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.173816 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-home\") pod \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " Sep 06 06:15:33.173899 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.173857 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-dshm\") pod \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " Sep 06 06:15:33.174115 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.173925 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5cdx\" (UniqueName: \"kubernetes.io/projected/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-kube-api-access-p5cdx\") pod \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\" (UID: \"2746cc5f-1b29-4182-a1b0-3c0ae93737e6\") " Sep 06 06:15:33.174115 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.174101 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2746cc5f-1b29-4182-a1b0-3c0ae93737e6" (UID: "2746cc5f-1b29-4182-a1b0-3c0ae93737e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:33.174224 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.174114 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-home" (OuterVolumeSpecName: "home") pod "2746cc5f-1b29-4182-a1b0-3c0ae93737e6" (UID: "2746cc5f-1b29-4182-a1b0-3c0ae93737e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:33.174224 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.174120 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "2746cc5f-1b29-4182-a1b0-3c0ae93737e6" (UID: "2746cc5f-1b29-4182-a1b0-3c0ae93737e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:33.174351 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.174330 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-home\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:33.174417 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.174353 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-model-cache\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:33.174417 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.174390 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-tmp-dir\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:33.176145 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.176116 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-dshm" (OuterVolumeSpecName: "dshm") pod "2746cc5f-1b29-4182-a1b0-3c0ae93737e6" (UID: "2746cc5f-1b29-4182-a1b0-3c0ae93737e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:33.176319 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.176300 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2746cc5f-1b29-4182-a1b0-3c0ae93737e6" (UID: "2746cc5f-1b29-4182-a1b0-3c0ae93737e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:15:33.176369 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.176325 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-kube-api-access-p5cdx" (OuterVolumeSpecName: "kube-api-access-p5cdx") pod "2746cc5f-1b29-4182-a1b0-3c0ae93737e6" (UID: "2746cc5f-1b29-4182-a1b0-3c0ae93737e6"). InnerVolumeSpecName "kube-api-access-p5cdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:15:33.275151 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.275127 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-dshm\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:33.275151 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.275148 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5cdx\" (UniqueName: \"kubernetes.io/projected/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-kube-api-access-p5cdx\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:33.275281 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.275159 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2746cc5f-1b29-4182-a1b0-3c0ae93737e6-tls-certs\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:33.616799 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.616766 2580 generic.go:356] "Generic (PLEG): container finished" podID="2746cc5f-1b29-4182-a1b0-3c0ae93737e6" containerID="5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918" exitCode=0 Sep 06 06:15:33.617034 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.616810 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" Sep 06 06:15:33.617034 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.616871 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" event={"ID":"2746cc5f-1b29-4182-a1b0-3c0ae93737e6","Type":"ContainerDied","Data":"5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918"} Sep 06 06:15:33.617034 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.616913 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw" event={"ID":"2746cc5f-1b29-4182-a1b0-3c0ae93737e6","Type":"ContainerDied","Data":"3c4624856503d70cc73cf3bd785feece683f9d470ffc7c71ac1ad7dae847f30f"} Sep 06 06:15:33.617034 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.616931 2580 scope.go:117] "RemoveContainer" containerID="5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918" Sep 06 06:15:33.627533 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.626761 2580 scope.go:117] "RemoveContainer" containerID="5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918" Sep 06 06:15:33.628167 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:15:33.628139 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918\": container with ID starting with 5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918 not found: ID does not exist" containerID="5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918" Sep 06 06:15:33.628275 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.628204 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918"} err="failed to get container status \"5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918\": rpc error: code = NotFound desc = could not find container \"5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918\": container with ID starting with 5d1e0c4d23a95a74d428dcdce7ffbd5a36a42e8056f3bc4e7a29b39d3b22b918 not found: ID does not exist" Sep 06 06:15:33.641032 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.641006 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw"] Sep 06 06:15:33.643148 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:33.643130 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0bce0b/test-model-0bce0b-kserve-666dbb4869-rfntw"] Sep 06 06:15:34.875741 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:34.875710 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2746cc5f-1b29-4182-a1b0-3c0ae93737e6" path="/var/lib/kubelet/pods/2746cc5f-1b29-4182-a1b0-3c0ae93737e6/volumes" Sep 06 06:15:46.078890 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.078850 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj"] Sep 06 06:15:46.081302 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.081277 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0bce0b-29811255-l2nsj"] Sep 06 06:15:46.103999 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.103972 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr"] Sep 06 06:15:46.104240 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.104215 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" podUID="b69a40a3-85f5-4c78-b9ca-8f8980025811" containerName="payload-processing" containerID="cri-o://08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f" gracePeriod=30 Sep 06 06:15:46.114398 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.114366 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv"] Sep 06 06:15:46.115107 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.114931 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" podUID="417b0b5f-25d5-4007-bdad-91493886b9d9" containerName="payload-pre-processing" containerID="cri-o://b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38" gracePeriod=30 Sep 06 06:15:46.386434 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.386400 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" Sep 06 06:15:46.389440 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.389415 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" Sep 06 06:15:46.499409 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.499379 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97gms\" (UniqueName: \"kubernetes.io/projected/b69a40a3-85f5-4c78-b9ca-8f8980025811-kube-api-access-97gms\") pod \"b69a40a3-85f5-4c78-b9ca-8f8980025811\" (UID: \"b69a40a3-85f5-4c78-b9ca-8f8980025811\") " Sep 06 06:15:46.499409 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.499409 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417b0b5f-25d5-4007-bdad-91493886b9d9-plugins-config-volume\") pod \"417b0b5f-25d5-4007-bdad-91493886b9d9\" (UID: \"417b0b5f-25d5-4007-bdad-91493886b9d9\") " Sep 06 06:15:46.499598 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.499443 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6tnz\" (UniqueName: \"kubernetes.io/projected/417b0b5f-25d5-4007-bdad-91493886b9d9-kube-api-access-s6tnz\") pod \"417b0b5f-25d5-4007-bdad-91493886b9d9\" (UID: \"417b0b5f-25d5-4007-bdad-91493886b9d9\") " Sep 06 06:15:46.499598 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.499460 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69a40a3-85f5-4c78-b9ca-8f8980025811-plugins-config-volume\") pod \"b69a40a3-85f5-4c78-b9ca-8f8980025811\" (UID: \"b69a40a3-85f5-4c78-b9ca-8f8980025811\") " Sep 06 06:15:46.499908 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.499813 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417b0b5f-25d5-4007-bdad-91493886b9d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "417b0b5f-25d5-4007-bdad-91493886b9d9" (UID: "417b0b5f-25d5-4007-bdad-91493886b9d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:15:46.500009 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.499886 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69a40a3-85f5-4c78-b9ca-8f8980025811-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b69a40a3-85f5-4c78-b9ca-8f8980025811" (UID: "b69a40a3-85f5-4c78-b9ca-8f8980025811"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:15:46.501547 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.501514 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69a40a3-85f5-4c78-b9ca-8f8980025811-kube-api-access-97gms" (OuterVolumeSpecName: "kube-api-access-97gms") pod "b69a40a3-85f5-4c78-b9ca-8f8980025811" (UID: "b69a40a3-85f5-4c78-b9ca-8f8980025811"). InnerVolumeSpecName "kube-api-access-97gms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:15:46.501547 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.501517 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417b0b5f-25d5-4007-bdad-91493886b9d9-kube-api-access-s6tnz" (OuterVolumeSpecName: "kube-api-access-s6tnz") pod "417b0b5f-25d5-4007-bdad-91493886b9d9" (UID: "417b0b5f-25d5-4007-bdad-91493886b9d9"). InnerVolumeSpecName "kube-api-access-s6tnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:15:46.601041 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.601016 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6tnz\" (UniqueName: \"kubernetes.io/projected/417b0b5f-25d5-4007-bdad-91493886b9d9-kube-api-access-s6tnz\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:46.601041 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.601038 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69a40a3-85f5-4c78-b9ca-8f8980025811-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:46.601218 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.601047 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97gms\" (UniqueName: \"kubernetes.io/projected/b69a40a3-85f5-4c78-b9ca-8f8980025811-kube-api-access-97gms\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:46.601218 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.601055 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417b0b5f-25d5-4007-bdad-91493886b9d9-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:46.703382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.703356 2580 generic.go:356] "Generic (PLEG): container finished" podID="417b0b5f-25d5-4007-bdad-91493886b9d9" containerID="b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38" exitCode=0 Sep 06 06:15:46.703530 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.703410 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" Sep 06 06:15:46.703530 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.703444 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" event={"ID":"417b0b5f-25d5-4007-bdad-91493886b9d9","Type":"ContainerDied","Data":"b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38"} Sep 06 06:15:46.703530 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.703500 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv" event={"ID":"417b0b5f-25d5-4007-bdad-91493886b9d9","Type":"ContainerDied","Data":"a894d52855e313d12818b5c9b70f14ebacd529895fde0095dfa0582bcfad9433"} Sep 06 06:15:46.703530 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.703523 2580 scope.go:117] "RemoveContainer" containerID="b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38" Sep 06 06:15:46.705080 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.705056 2580 generic.go:356] "Generic (PLEG): container finished" podID="b69a40a3-85f5-4c78-b9ca-8f8980025811" containerID="08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f" exitCode=0 Sep 06 06:15:46.705189 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.705109 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" event={"ID":"b69a40a3-85f5-4c78-b9ca-8f8980025811","Type":"ContainerDied","Data":"08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f"} Sep 06 06:15:46.705189 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.705117 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" Sep 06 06:15:46.705189 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.705146 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr" event={"ID":"b69a40a3-85f5-4c78-b9ca-8f8980025811","Type":"ContainerDied","Data":"c7e60595f6fc6abf67ba847554da3628c79a0537504a91c8f7d1b67c0f4b3bee"} Sep 06 06:15:46.713215 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.713126 2580 scope.go:117] "RemoveContainer" containerID="b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38" Sep 06 06:15:46.713554 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:15:46.713532 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38\": container with ID starting with b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38 not found: ID does not exist" containerID="b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38" Sep 06 06:15:46.713656 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.713564 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38"} err="failed to get container status \"b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38\": rpc error: code = NotFound desc = could not find container \"b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38\": container with ID starting with b8f428d69d47cc90d29dc14f66f3200368eb7424e45d6a3a36e2806381c7cf38 not found: ID does not exist" Sep 06 06:15:46.713656 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.713587 2580 scope.go:117] "RemoveContainer" containerID="08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f" Sep 06 06:15:46.724887 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.724802 2580 scope.go:117] "RemoveContainer" containerID="08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f" Sep 06 06:15:46.725354 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:15:46.725335 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f\": container with ID starting with 08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f not found: ID does not exist" containerID="08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f" Sep 06 06:15:46.725438 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.725362 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f"} err="failed to get container status \"08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f\": rpc error: code = NotFound desc = could not find container \"08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f\": container with ID starting with 08aad06a22275b70ac4b5f968ae07d18ebbf96065160fbb79b87317e066d684f not found: ID does not exist" Sep 06 06:15:46.732141 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.732116 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv"] Sep 06 06:15:46.734703 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.734679 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0bce0b-7c6d4bbb4b-skntv"] Sep 06 06:15:46.745210 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.745188 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr"] Sep 06 06:15:46.749971 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.749950 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0bce0b-7dcccd6f59-zcsvr"] Sep 06 06:15:46.752853 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:15:46.752816 2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69a40a3_85f5_4c78_b9ca_8f8980025811.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417b0b5f_25d5_4007_bdad_91493886b9d9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417b0b5f_25d5_4007_bdad_91493886b9d9.slice/crio-a894d52855e313d12818b5c9b70f14ebacd529895fde0095dfa0582bcfad9433\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69a40a3_85f5_4c78_b9ca_8f8980025811.slice/crio-c7e60595f6fc6abf67ba847554da3628c79a0537504a91c8f7d1b67c0f4b3bee\": RecentStats: unable to find data in memory cache]" Sep 06 06:15:46.875275 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.875252 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="138423f6-da26-4506-956f-4033318f776a" path="/var/lib/kubelet/pods/138423f6-da26-4506-956f-4033318f776a/volumes" Sep 06 06:15:46.875555 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.875543 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="417b0b5f-25d5-4007-bdad-91493886b9d9" path="/var/lib/kubelet/pods/417b0b5f-25d5-4007-bdad-91493886b9d9/volumes" Sep 06 06:15:46.875903 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:46.875890 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b69a40a3-85f5-4c78-b9ca-8f8980025811" path="/var/lib/kubelet/pods/b69a40a3-85f5-4c78-b9ca-8f8980025811/volumes" Sep 06 06:15:49.970660 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:49.970623 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt"] Sep 06 06:15:49.971063 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:49.970916 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" podUID="2de7ae89-45d2-44ee-b8b4-e8cf473f09d8" containerName="payload-processing" containerID="cri-o://24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7" gracePeriod=30 Sep 06 06:15:50.220870 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.220787 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" Sep 06 06:15:50.333937 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.333912 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6psfn\" (UniqueName: \"kubernetes.io/projected/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8-kube-api-access-6psfn\") pod \"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8\" (UID: \"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8\") " Sep 06 06:15:50.334090 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.333960 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8-plugins-config-volume\") pod \"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8\" (UID: \"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8\") " Sep 06 06:15:50.334374 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.334352 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2de7ae89-45d2-44ee-b8b4-e8cf473f09d8" (UID: "2de7ae89-45d2-44ee-b8b4-e8cf473f09d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:15:50.335956 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.335921 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8-kube-api-access-6psfn" (OuterVolumeSpecName: "kube-api-access-6psfn") pod "2de7ae89-45d2-44ee-b8b4-e8cf473f09d8" (UID: "2de7ae89-45d2-44ee-b8b4-e8cf473f09d8"). InnerVolumeSpecName "kube-api-access-6psfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:15:50.435074 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.435045 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6psfn\" (UniqueName: \"kubernetes.io/projected/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8-kube-api-access-6psfn\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:50.435074 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.435069 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:15:50.729628 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.729596 2580 generic.go:356] "Generic (PLEG): container finished" podID="2de7ae89-45d2-44ee-b8b4-e8cf473f09d8" containerID="24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7" exitCode=0 Sep 06 06:15:50.729784 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.729642 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" Sep 06 06:15:50.729784 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.729636 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" event={"ID":"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8","Type":"ContainerDied","Data":"24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7"} Sep 06 06:15:50.729784 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.729760 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt" event={"ID":"2de7ae89-45d2-44ee-b8b4-e8cf473f09d8","Type":"ContainerDied","Data":"1e30249b4177caa3c57c31e4267aa59f4d7f1b741e59d95647d6e31766d27b56"} Sep 06 06:15:50.729784 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.729783 2580 scope.go:117] "RemoveContainer" containerID="24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7" Sep 06 06:15:50.738435 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.738425 2580 scope.go:117] "RemoveContainer" containerID="24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7" Sep 06 06:15:50.738690 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:15:50.738673 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7\": container with ID starting with 24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7 not found: ID does not exist" containerID="24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7" Sep 06 06:15:50.738738 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.738695 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7"} err="failed to get container status \"24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7\": rpc error: code = NotFound desc = could not find container \"24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7\": container with ID starting with 24dc70970db3c835eeb3d95d48dbe47fefb0664385e924d946fff39b62931fe7 not found: ID does not exist" Sep 06 06:15:50.750252 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.750233 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt"] Sep 06 06:15:50.753703 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.753682 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc95779d-7b59f86649-nvpqt"] Sep 06 06:15:50.876443 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:15:50.876414 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2de7ae89-45d2-44ee-b8b4-e8cf473f09d8" path="/var/lib/kubelet/pods/2de7ae89-45d2-44ee-b8b4-e8cf473f09d8/volumes" Sep 06 06:16:14.633812 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.633773 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f"] Sep 06 06:16:14.634429 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634412 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69a40a3-85f5-4c78-b9ca-8f8980025811" containerName="payload-processing" Sep 06 06:16:14.634476 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634431 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69a40a3-85f5-4c78-b9ca-8f8980025811" containerName="payload-processing" Sep 06 06:16:14.634476 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634446 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2de7ae89-45d2-44ee-b8b4-e8cf473f09d8" containerName="payload-processing" Sep 06 06:16:14.634476 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634455 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2de7ae89-45d2-44ee-b8b4-e8cf473f09d8" containerName="payload-processing" Sep 06 06:16:14.634476 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634470 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2746cc5f-1b29-4182-a1b0-3c0ae93737e6" containerName="main" Sep 06 06:16:14.634602 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634479 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2746cc5f-1b29-4182-a1b0-3c0ae93737e6" containerName="main" Sep 06 06:16:14.634602 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634491 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="417b0b5f-25d5-4007-bdad-91493886b9d9" containerName="payload-pre-processing" Sep 06 06:16:14.634602 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634499 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="417b0b5f-25d5-4007-bdad-91493886b9d9" containerName="payload-pre-processing" Sep 06 06:16:14.634602 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634520 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="709f0807-b7ad-489d-82ad-6591f16dad8f" containerName="main" Sep 06 06:16:14.634602 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634528 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="709f0807-b7ad-489d-82ad-6591f16dad8f" containerName="main" Sep 06 06:16:14.634750 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634631 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="417b0b5f-25d5-4007-bdad-91493886b9d9" containerName="payload-pre-processing" Sep 06 06:16:14.634750 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634645 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b69a40a3-85f5-4c78-b9ca-8f8980025811" containerName="payload-processing" Sep 06 06:16:14.634750 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634655 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="709f0807-b7ad-489d-82ad-6591f16dad8f" containerName="main" Sep 06 06:16:14.634750 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634664 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2746cc5f-1b29-4182-a1b0-3c0ae93737e6" containerName="main" Sep 06 06:16:14.634750 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.634675 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2de7ae89-45d2-44ee-b8b4-e8cf473f09d8" containerName="payload-processing" Sep 06 06:16:14.639702 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.639680 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" Sep 06 06:16:14.642147 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.642128 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-340e00-dockercfg-pqz78\"" Sep 06 06:16:14.642244 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.642147 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-340e00\"" Sep 06 06:16:14.647409 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.647234 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f"] Sep 06 06:16:14.649804 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.649779 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f"] Sep 06 06:16:14.653879 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.653860 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" Sep 06 06:16:14.661211 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.661191 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f"] Sep 06 06:16:14.744399 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.744373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4rcn\" (UniqueName: \"kubernetes.io/projected/25372e71-56b0-4221-bb07-5d1fddc4fdda-kube-api-access-h4rcn\") pod \"payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f\" (UID: \"25372e71-56b0-4221-bb07-5d1fddc4fdda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" Sep 06 06:16:14.744523 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.744416 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25372e71-56b0-4221-bb07-5d1fddc4fdda-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f\" (UID: \"25372e71-56b0-4221-bb07-5d1fddc4fdda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" Sep 06 06:16:14.744523 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.744471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x44d\" (UniqueName: \"kubernetes.io/projected/30148597-c0f0-4d25-954f-2cd5db81152c-kube-api-access-5x44d\") pod \"payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f\" (UID: \"30148597-c0f0-4d25-954f-2cd5db81152c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" Sep 06 06:16:14.744603 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.744532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30148597-c0f0-4d25-954f-2cd5db81152c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f\" (UID: \"30148597-c0f0-4d25-954f-2cd5db81152c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" Sep 06 06:16:14.845500 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.845469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30148597-c0f0-4d25-954f-2cd5db81152c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f\" (UID: \"30148597-c0f0-4d25-954f-2cd5db81152c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" Sep 06 06:16:14.845635 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.845541 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4rcn\" (UniqueName: \"kubernetes.io/projected/25372e71-56b0-4221-bb07-5d1fddc4fdda-kube-api-access-h4rcn\") pod \"payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f\" (UID: \"25372e71-56b0-4221-bb07-5d1fddc4fdda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" Sep 06 06:16:14.845635 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.845576 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25372e71-56b0-4221-bb07-5d1fddc4fdda-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f\" (UID: \"25372e71-56b0-4221-bb07-5d1fddc4fdda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" Sep 06 06:16:14.845635 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.845604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x44d\" (UniqueName: \"kubernetes.io/projected/30148597-c0f0-4d25-954f-2cd5db81152c-kube-api-access-5x44d\") pod \"payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f\" (UID: \"30148597-c0f0-4d25-954f-2cd5db81152c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" Sep 06 06:16:14.846242 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.846220 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30148597-c0f0-4d25-954f-2cd5db81152c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f\" (UID: \"30148597-c0f0-4d25-954f-2cd5db81152c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" Sep 06 06:16:14.846363 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.846336 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25372e71-56b0-4221-bb07-5d1fddc4fdda-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f\" (UID: \"25372e71-56b0-4221-bb07-5d1fddc4fdda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" Sep 06 06:16:14.853738 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.853709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4rcn\" (UniqueName: \"kubernetes.io/projected/25372e71-56b0-4221-bb07-5d1fddc4fdda-kube-api-access-h4rcn\") pod \"payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f\" (UID: \"25372e71-56b0-4221-bb07-5d1fddc4fdda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" Sep 06 06:16:14.854249 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.854227 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x44d\" (UniqueName: \"kubernetes.io/projected/30148597-c0f0-4d25-954f-2cd5db81152c-kube-api-access-5x44d\") pod \"payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f\" (UID: \"30148597-c0f0-4d25-954f-2cd5db81152c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" Sep 06 06:16:14.951426 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.951400 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" Sep 06 06:16:14.965568 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:14.965547 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" Sep 06 06:16:15.225940 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:15.225911 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f"] Sep 06 06:16:15.227131 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:16:15.227100 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30148597_c0f0_4d25_954f_2cd5db81152c.slice/crio-d385aacd9f1c6873a6e3dbaa29e7597430211274d1627c054cedeecf55267b2e WatchSource:0}: Error finding container d385aacd9f1c6873a6e3dbaa29e7597430211274d1627c054cedeecf55267b2e: Status 404 returned error can't find the container with id d385aacd9f1c6873a6e3dbaa29e7597430211274d1627c054cedeecf55267b2e Sep 06 06:16:15.245769 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:15.245745 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f"] Sep 06 06:16:15.247475 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:16:15.247449 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25372e71_56b0_4221_bb07_5d1fddc4fdda.slice/crio-595c7dc551bd69884cf2274b20cd231720a31114340db4f9cc1cc2c777904954 WatchSource:0}: Error finding container 595c7dc551bd69884cf2274b20cd231720a31114340db4f9cc1cc2c777904954: Status 404 returned error can't find the container with id 595c7dc551bd69884cf2274b20cd231720a31114340db4f9cc1cc2c777904954 Sep 06 06:16:15.855638 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:15.855602 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" event={"ID":"25372e71-56b0-4221-bb07-5d1fddc4fdda","Type":"ContainerStarted","Data":"d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b"} Sep 06 06:16:15.856123 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:15.855666 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" event={"ID":"25372e71-56b0-4221-bb07-5d1fddc4fdda","Type":"ContainerStarted","Data":"595c7dc551bd69884cf2274b20cd231720a31114340db4f9cc1cc2c777904954"} Sep 06 06:16:15.856123 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:15.855699 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" Sep 06 06:16:15.857386 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:15.857356 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" event={"ID":"30148597-c0f0-4d25-954f-2cd5db81152c","Type":"ContainerStarted","Data":"7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb"} Sep 06 06:16:15.857510 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:15.857415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" event={"ID":"30148597-c0f0-4d25-954f-2cd5db81152c","Type":"ContainerStarted","Data":"d385aacd9f1c6873a6e3dbaa29e7597430211274d1627c054cedeecf55267b2e"} Sep 06 06:16:15.857510 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:15.857449 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" Sep 06 06:16:15.872930 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:15.872876 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" podStartSLOduration=1.8728571 podStartE2EDuration="1.8728571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:16:15.870897991 +0000 UTC m=+1341.519517171" watchObservedRunningTime="2026-09-06 06:16:15.8728571 +0000 UTC m=+1341.521476282" Sep 06 06:16:15.886548 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:15.886509 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" podStartSLOduration=1.8864974220000001 podStartE2EDuration="1.886497422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:16:15.885181651 +0000 UTC m=+1341.533800830" watchObservedRunningTime="2026-09-06 06:16:15.886497422 +0000 UTC m=+1341.535116627" Sep 06 06:16:26.864714 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:26.864630 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" Sep 06 06:16:26.864714 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:26.864710 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" Sep 06 06:16:41.494278 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.494236 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g"] Sep 06 06:16:41.497921 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.497898 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" Sep 06 06:16:41.500695 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.500610 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-dc97f9-dockercfg-jg27p\"" Sep 06 06:16:41.500960 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.500942 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-dc97f9\"" Sep 06 06:16:41.508786 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.508761 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g"] Sep 06 06:16:41.519292 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.519269 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx"] Sep 06 06:16:41.523070 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.523053 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" Sep 06 06:16:41.532434 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.532415 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx"] Sep 06 06:16:41.599305 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.599277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84108c7d-4f79-4ec5-bd64-93cb13d1e051-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g\" (UID: \"84108c7d-4f79-4ec5-bd64-93cb13d1e051\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" Sep 06 06:16:41.599432 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.599331 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzcv2\" (UniqueName: \"kubernetes.io/projected/84108c7d-4f79-4ec5-bd64-93cb13d1e051-kube-api-access-pzcv2\") pod \"payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g\" (UID: \"84108c7d-4f79-4ec5-bd64-93cb13d1e051\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" Sep 06 06:16:41.700531 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.700505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8sc\" (UniqueName: \"kubernetes.io/projected/85a9d189-2ffa-4d9e-bfac-a29c14508d64-kube-api-access-9f8sc\") pod \"payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx\" (UID: \"85a9d189-2ffa-4d9e-bfac-a29c14508d64\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" Sep 06 06:16:41.700659 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.700545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84108c7d-4f79-4ec5-bd64-93cb13d1e051-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g\" (UID: \"84108c7d-4f79-4ec5-bd64-93cb13d1e051\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" Sep 06 06:16:41.700659 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.700565 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a9d189-2ffa-4d9e-bfac-a29c14508d64-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx\" (UID: \"85a9d189-2ffa-4d9e-bfac-a29c14508d64\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" Sep 06 06:16:41.700742 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.700659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzcv2\" (UniqueName: \"kubernetes.io/projected/84108c7d-4f79-4ec5-bd64-93cb13d1e051-kube-api-access-pzcv2\") pod \"payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g\" (UID: \"84108c7d-4f79-4ec5-bd64-93cb13d1e051\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" Sep 06 06:16:41.701214 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.701196 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84108c7d-4f79-4ec5-bd64-93cb13d1e051-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g\" (UID: \"84108c7d-4f79-4ec5-bd64-93cb13d1e051\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" Sep 06 06:16:41.713131 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.713106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzcv2\" (UniqueName: \"kubernetes.io/projected/84108c7d-4f79-4ec5-bd64-93cb13d1e051-kube-api-access-pzcv2\") pod \"payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g\" (UID: \"84108c7d-4f79-4ec5-bd64-93cb13d1e051\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" Sep 06 06:16:41.802154 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.802101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8sc\" (UniqueName: \"kubernetes.io/projected/85a9d189-2ffa-4d9e-bfac-a29c14508d64-kube-api-access-9f8sc\") pod \"payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx\" (UID: \"85a9d189-2ffa-4d9e-bfac-a29c14508d64\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" Sep 06 06:16:41.802154 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.802135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a9d189-2ffa-4d9e-bfac-a29c14508d64-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx\" (UID: \"85a9d189-2ffa-4d9e-bfac-a29c14508d64\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" Sep 06 06:16:41.802706 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.802689 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a9d189-2ffa-4d9e-bfac-a29c14508d64-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx\" (UID: \"85a9d189-2ffa-4d9e-bfac-a29c14508d64\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" Sep 06 06:16:41.809367 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.809338 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8sc\" (UniqueName: \"kubernetes.io/projected/85a9d189-2ffa-4d9e-bfac-a29c14508d64-kube-api-access-9f8sc\") pod \"payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx\" (UID: \"85a9d189-2ffa-4d9e-bfac-a29c14508d64\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" Sep 06 06:16:41.810783 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.810760 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" Sep 06 06:16:41.832796 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:41.832775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" Sep 06 06:16:42.084760 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:42.084734 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g"] Sep 06 06:16:42.086219 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:16:42.086192 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84108c7d_4f79_4ec5_bd64_93cb13d1e051.slice/crio-a526f096e511862659eed253cc27af3c28f9f10f32872f71174dd6be2b6adade WatchSource:0}: Error finding container a526f096e511862659eed253cc27af3c28f9f10f32872f71174dd6be2b6adade: Status 404 returned error can't find the container with id a526f096e511862659eed253cc27af3c28f9f10f32872f71174dd6be2b6adade Sep 06 06:16:42.117716 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:42.117693 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx"] Sep 06 06:16:42.118956 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:16:42.118933 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a9d189_2ffa_4d9e_bfac_a29c14508d64.slice/crio-18d66f664f1a8dcc3915df60404fa5656c4cc85a607a029a801d5468a40c2c4e WatchSource:0}: Error finding container 18d66f664f1a8dcc3915df60404fa5656c4cc85a607a029a801d5468a40c2c4e: Status 404 returned error can't find the container with id 18d66f664f1a8dcc3915df60404fa5656c4cc85a607a029a801d5468a40c2c4e Sep 06 06:16:43.002625 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.002586 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" event={"ID":"85a9d189-2ffa-4d9e-bfac-a29c14508d64","Type":"ContainerStarted","Data":"b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc"} Sep 06 06:16:43.003084 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.002682 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" Sep 06 06:16:43.003084 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.002709 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" event={"ID":"85a9d189-2ffa-4d9e-bfac-a29c14508d64","Type":"ContainerStarted","Data":"18d66f664f1a8dcc3915df60404fa5656c4cc85a607a029a801d5468a40c2c4e"} Sep 06 06:16:43.004300 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.004271 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" event={"ID":"84108c7d-4f79-4ec5-bd64-93cb13d1e051","Type":"ContainerStarted","Data":"0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a"} Sep 06 06:16:43.004447 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.004319 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" event={"ID":"84108c7d-4f79-4ec5-bd64-93cb13d1e051","Type":"ContainerStarted","Data":"a526f096e511862659eed253cc27af3c28f9f10f32872f71174dd6be2b6adade"} Sep 06 06:16:43.004447 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.004345 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" Sep 06 06:16:43.022881 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.022815 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" podStartSLOduration=2.022797693 podStartE2EDuration="2.022797693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:16:43.021038578 +0000 UTC m=+1368.669657760" watchObservedRunningTime="2026-09-06 06:16:43.022797693 +0000 UTC m=+1368.671416872" Sep 06 06:16:43.037468 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.037427 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" podStartSLOduration=2.037414264 podStartE2EDuration="2.037414264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:16:43.036767112 +0000 UTC m=+1368.685386292" watchObservedRunningTime="2026-09-06 06:16:43.037414264 +0000 UTC m=+1368.686033442" Sep 06 06:16:43.797889 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.797857 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp"] Sep 06 06:16:43.801121 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.801103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" Sep 06 06:16:43.804553 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.804503 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b7c728-dockercfg-hn85c\"" Sep 06 06:16:43.804553 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.804560 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b7c728\"" Sep 06 06:16:43.814197 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.814171 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp"] Sep 06 06:16:43.834758 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.834729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp\" (UID: \"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" Sep 06 06:16:43.834876 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.834842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg89j\" (UniqueName: \"kubernetes.io/projected/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3-kube-api-access-hg89j\") pod \"payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp\" (UID: \"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" Sep 06 06:16:43.842555 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.842528 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv"] Sep 06 06:16:43.846017 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.845994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" Sep 06 06:16:43.855334 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.855310 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv"] Sep 06 06:16:43.935477 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.935452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg89j\" (UniqueName: \"kubernetes.io/projected/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3-kube-api-access-hg89j\") pod \"payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp\" (UID: \"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" Sep 06 06:16:43.935606 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.935496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gxqz\" (UniqueName: \"kubernetes.io/projected/e2229fad-7b16-4b47-96ce-261750fc7470-kube-api-access-2gxqz\") pod \"payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv\" (UID: \"e2229fad-7b16-4b47-96ce-261750fc7470\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" Sep 06 06:16:43.935691 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.935632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp\" (UID: \"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" Sep 06 06:16:43.935691 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.935684 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2229fad-7b16-4b47-96ce-261750fc7470-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv\" (UID: \"e2229fad-7b16-4b47-96ce-261750fc7470\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" Sep 06 06:16:43.936268 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.936250 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp\" (UID: \"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" Sep 06 06:16:43.942959 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:43.942935 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg89j\" (UniqueName: \"kubernetes.io/projected/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3-kube-api-access-hg89j\") pod \"payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp\" (UID: \"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" Sep 06 06:16:44.036180 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:44.036154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2229fad-7b16-4b47-96ce-261750fc7470-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv\" (UID: \"e2229fad-7b16-4b47-96ce-261750fc7470\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" Sep 06 06:16:44.036526 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:44.036268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gxqz\" (UniqueName: \"kubernetes.io/projected/e2229fad-7b16-4b47-96ce-261750fc7470-kube-api-access-2gxqz\") pod \"payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv\" (UID: \"e2229fad-7b16-4b47-96ce-261750fc7470\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" Sep 06 06:16:44.036991 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:44.036969 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2229fad-7b16-4b47-96ce-261750fc7470-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv\" (UID: \"e2229fad-7b16-4b47-96ce-261750fc7470\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" Sep 06 06:16:44.047494 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:44.047468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gxqz\" (UniqueName: \"kubernetes.io/projected/e2229fad-7b16-4b47-96ce-261750fc7470-kube-api-access-2gxqz\") pod \"payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv\" (UID: \"e2229fad-7b16-4b47-96ce-261750fc7470\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" Sep 06 06:16:44.113724 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:44.113677 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" Sep 06 06:16:44.155967 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:44.155940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" Sep 06 06:16:44.386133 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:44.386106 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp"] Sep 06 06:16:44.387534 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:16:44.387503 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d5b4e3_22af_45c9_9ec0_2df27edcb4b3.slice/crio-5364d863a1dfa9ffaea6968236b47eb9cf2a1318ee30579a5ce050d4e4118da6 WatchSource:0}: Error finding container 5364d863a1dfa9ffaea6968236b47eb9cf2a1318ee30579a5ce050d4e4118da6: Status 404 returned error can't find the container with id 5364d863a1dfa9ffaea6968236b47eb9cf2a1318ee30579a5ce050d4e4118da6 Sep 06 06:16:44.437782 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:44.437761 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv"] Sep 06 06:16:44.440148 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:16:44.440124 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2229fad_7b16_4b47_96ce_261750fc7470.slice/crio-816745d74e6f7539e0dc3c8b19ec35d71cb8dc6814b33cd150741c4332ca7a18 WatchSource:0}: Error finding container 816745d74e6f7539e0dc3c8b19ec35d71cb8dc6814b33cd150741c4332ca7a18: Status 404 returned error can't find the container with id 816745d74e6f7539e0dc3c8b19ec35d71cb8dc6814b33cd150741c4332ca7a18 Sep 06 06:16:45.017329 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:45.017293 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" event={"ID":"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3","Type":"ContainerStarted","Data":"2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158"} Sep 06 06:16:45.017543 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:45.017353 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" event={"ID":"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3","Type":"ContainerStarted","Data":"5364d863a1dfa9ffaea6968236b47eb9cf2a1318ee30579a5ce050d4e4118da6"} Sep 06 06:16:45.017543 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:45.017384 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" Sep 06 06:16:45.018961 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:45.018929 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" event={"ID":"e2229fad-7b16-4b47-96ce-261750fc7470","Type":"ContainerStarted","Data":"df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244"} Sep 06 06:16:45.019084 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:45.018985 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" Sep 06 06:16:45.019084 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:45.019009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" event={"ID":"e2229fad-7b16-4b47-96ce-261750fc7470","Type":"ContainerStarted","Data":"816745d74e6f7539e0dc3c8b19ec35d71cb8dc6814b33cd150741c4332ca7a18"} Sep 06 06:16:45.034130 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:45.034082 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" podStartSLOduration=2.034068857 podStartE2EDuration="2.034068857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:16:45.032730111 +0000 UTC m=+1370.681349291" watchObservedRunningTime="2026-09-06 06:16:45.034068857 +0000 UTC m=+1370.682688037" Sep 06 06:16:45.049115 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:45.049073 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" podStartSLOduration=2.049060805 podStartE2EDuration="2.049060805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:16:45.047070581 +0000 UTC m=+1370.695689761" watchObservedRunningTime="2026-09-06 06:16:45.049060805 +0000 UTC m=+1370.697679985" Sep 06 06:16:54.011516 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:54.011485 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" Sep 06 06:16:54.012023 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:54.011658 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" Sep 06 06:16:56.026266 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:56.026228 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" Sep 06 06:16:56.026707 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:16:56.026546 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" Sep 06 06:17:12.402662 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.402623 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd"] Sep 06 06:17:12.406854 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.406819 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.411558 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.411535 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-429831\"/\"kube-root-ca.crt\"" Sep 06 06:17:12.411716 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.411694 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-429831\"/\"default-dockercfg-2vjht\"" Sep 06 06:17:12.411863 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.411628 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-429831\"/\"auth-test-model-429831-kserve-self-signed-certs\"" Sep 06 06:17:12.411948 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.411631 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-429831\"/\"openshift-service-ca.crt\"" Sep 06 06:17:12.414029 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.414010 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd"] Sep 06 06:17:12.486820 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.486792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb20345-c78d-490b-9a29-4e1433e5f190-tls-certs\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.486966 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.486861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-home\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.486966 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.486926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-model-cache\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.486966 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.486953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjm4k\" (UniqueName: \"kubernetes.io/projected/5cb20345-c78d-490b-9a29-4e1433e5f190-kube-api-access-zjm4k\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.487125 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.486976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-tmp-dir\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.487125 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.487046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-dshm\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.587496 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.587471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-model-cache\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.587619 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.587503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjm4k\" (UniqueName: \"kubernetes.io/projected/5cb20345-c78d-490b-9a29-4e1433e5f190-kube-api-access-zjm4k\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.587619 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.587525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-tmp-dir\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.587619 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.587551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-dshm\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.587619 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.587611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb20345-c78d-490b-9a29-4e1433e5f190-tls-certs\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.587861 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.587644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-home\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.587931 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.587880 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-model-cache\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.587984 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.587965 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-tmp-dir\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.588020 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.587982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-home\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.589958 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.589934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-dshm\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.590224 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.590203 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb20345-c78d-490b-9a29-4e1433e5f190-tls-certs\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.597787 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.597767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjm4k\" (UniqueName: \"kubernetes.io/projected/5cb20345-c78d-490b-9a29-4e1433e5f190-kube-api-access-zjm4k\") pod \"auth-test-model-429831-kserve-566b5fb6ff-7tdjd\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.717928 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.717905 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:12.988955 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:12.988927 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd"] Sep 06 06:17:12.990571 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:17:12.990540 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cb20345_c78d_490b_9a29_4e1433e5f190.slice/crio-b4c54aba571c5d9b213d3acca220e4580f6bc58473e8c356c544da1d1e8d3351 WatchSource:0}: Error finding container b4c54aba571c5d9b213d3acca220e4580f6bc58473e8c356c544da1d1e8d3351: Status 404 returned error can't find the container with id b4c54aba571c5d9b213d3acca220e4580f6bc58473e8c356c544da1d1e8d3351 Sep 06 06:17:13.160286 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:13.160253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" event={"ID":"5cb20345-c78d-490b-9a29-4e1433e5f190","Type":"ContainerStarted","Data":"3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630"} Sep 06 06:17:13.160446 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:13.160295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" event={"ID":"5cb20345-c78d-490b-9a29-4e1433e5f190","Type":"ContainerStarted","Data":"b4c54aba571c5d9b213d3acca220e4580f6bc58473e8c356c544da1d1e8d3351"} Sep 06 06:17:13.175780 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:13.175730 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" podStartSLOduration=1.175714562 podStartE2EDuration="1.175714562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:17:13.17524019 +0000 UTC m=+1398.823859371" watchObservedRunningTime="2026-09-06 06:17:13.175714562 +0000 UTC m=+1398.824333741" Sep 06 06:17:22.718033 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:22.717989 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:22.718033 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:22.718044 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:22.730358 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:22.730331 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:23.222326 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:23.222288 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:17:24.796476 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:24.796442 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np"] Sep 06 06:17:24.800084 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:24.800068 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" Sep 06 06:17:24.802951 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:24.802928 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-dd8c231a\"" Sep 06 06:17:24.812583 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:24.812558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np"] Sep 06 06:17:24.901043 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:24.901005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z94bj\" (UniqueName: \"kubernetes.io/projected/b86441ec-02e4-408a-9ae9-f65ee084dc9f-kube-api-access-z94bj\") pod \"payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np\" (UID: \"b86441ec-02e4-408a-9ae9-f65ee084dc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" Sep 06 06:17:24.901203 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:24.901136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86441ec-02e4-408a-9ae9-f65ee084dc9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np\" (UID: \"b86441ec-02e4-408a-9ae9-f65ee084dc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" Sep 06 06:17:25.002639 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:25.002604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z94bj\" (UniqueName: \"kubernetes.io/projected/b86441ec-02e4-408a-9ae9-f65ee084dc9f-kube-api-access-z94bj\") pod \"payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np\" (UID: \"b86441ec-02e4-408a-9ae9-f65ee084dc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" Sep 06 06:17:25.002889 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:25.002859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86441ec-02e4-408a-9ae9-f65ee084dc9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np\" (UID: \"b86441ec-02e4-408a-9ae9-f65ee084dc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" Sep 06 06:17:25.003596 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:25.003573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86441ec-02e4-408a-9ae9-f65ee084dc9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np\" (UID: \"b86441ec-02e4-408a-9ae9-f65ee084dc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" Sep 06 06:17:25.010425 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:25.010401 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z94bj\" (UniqueName: \"kubernetes.io/projected/b86441ec-02e4-408a-9ae9-f65ee084dc9f-kube-api-access-z94bj\") pod \"payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np\" (UID: \"b86441ec-02e4-408a-9ae9-f65ee084dc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" Sep 06 06:17:25.111474 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:25.111411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" Sep 06 06:17:25.387352 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:25.387321 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np"] Sep 06 06:17:25.389146 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:17:25.389115 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86441ec_02e4_408a_9ae9_f65ee084dc9f.slice/crio-f12dbcbc467b5ea3bbe1c82e16cba313292a4d3ebace157b91bb15734da4955b WatchSource:0}: Error finding container f12dbcbc467b5ea3bbe1c82e16cba313292a4d3ebace157b91bb15734da4955b: Status 404 returned error can't find the container with id f12dbcbc467b5ea3bbe1c82e16cba313292a4d3ebace157b91bb15734da4955b Sep 06 06:17:26.228607 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:26.228568 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" event={"ID":"b86441ec-02e4-408a-9ae9-f65ee084dc9f","Type":"ContainerStarted","Data":"74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde"} Sep 06 06:17:26.228993 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:26.228630 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" event={"ID":"b86441ec-02e4-408a-9ae9-f65ee084dc9f","Type":"ContainerStarted","Data":"f12dbcbc467b5ea3bbe1c82e16cba313292a4d3ebace157b91bb15734da4955b"} Sep 06 06:17:26.228993 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:26.228730 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" Sep 06 06:17:26.245444 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:26.245396 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" podStartSLOduration=2.245382782 podStartE2EDuration="2.245382782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:17:26.243868536 +0000 UTC m=+1411.892487711" watchObservedRunningTime="2026-09-06 06:17:26.245382782 +0000 UTC m=+1411.894001981" Sep 06 06:17:37.236506 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:17:37.236475 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" Sep 06 06:18:24.706256 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.706221 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g"] Sep 06 06:18:24.710943 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.710922 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.714638 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.714612 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dd8c231a\"/\"e2e-lifecycle-model-dd8c231a-kserve-self-signed-certs\"" Sep 06 06:18:24.714750 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.714619 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dd8c231a\"/\"default-dockercfg-8859z\"" Sep 06 06:18:24.714750 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.714623 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dd8c231a\"/\"openshift-service-ca.crt\"" Sep 06 06:18:24.714750 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.714708 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dd8c231a\"/\"kube-root-ca.crt\"" Sep 06 06:18:24.720542 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.720496 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g"] Sep 06 06:18:24.822906 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.822863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-home\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.823038 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.822912 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-dshm\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.823038 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.822975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-model-cache\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.823112 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.823048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkr58\" (UniqueName: \"kubernetes.io/projected/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-kube-api-access-pkr58\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.823112 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.823101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-tmp-dir\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.823181 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.823148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-tls-certs\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.923726 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.923695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-tmp-dir\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.923919 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.923758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-tls-certs\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.923919 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.923816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-home\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.923919 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.923871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-dshm\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.923919 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.923906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-model-cache\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.924140 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.924043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkr58\" (UniqueName: \"kubernetes.io/projected/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-kube-api-access-pkr58\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.924198 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.924147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-tmp-dir\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.924198 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.924163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-home\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.924300 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.924252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-model-cache\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.926084 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.926068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-dshm\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.926183 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.926167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-tls-certs\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:24.932303 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:24.932286 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkr58\" (UniqueName: \"kubernetes.io/projected/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-kube-api-access-pkr58\") pod \"e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:25.022379 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:25.022318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:25.292240 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:25.292207 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g"] Sep 06 06:18:25.293400 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:18:25.293374 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cd62a68_f0f9_48bf_ac4f_2f6850572a84.slice/crio-2e6f64cf8f90eab0d28fccd6802104df9de3336e06f964525bc17535cbf3d6b7 WatchSource:0}: Error finding container 2e6f64cf8f90eab0d28fccd6802104df9de3336e06f964525bc17535cbf3d6b7: Status 404 returned error can't find the container with id 2e6f64cf8f90eab0d28fccd6802104df9de3336e06f964525bc17535cbf3d6b7 Sep 06 06:18:25.546328 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:25.546212 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" event={"ID":"6cd62a68-f0f9-48bf-ac4f-2f6850572a84","Type":"ContainerStarted","Data":"3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1"} Sep 06 06:18:25.546328 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:25.546261 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" event={"ID":"6cd62a68-f0f9-48bf-ac4f-2f6850572a84","Type":"ContainerStarted","Data":"2e6f64cf8f90eab0d28fccd6802104df9de3336e06f964525bc17535cbf3d6b7"} Sep 06 06:18:25.561898 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:25.561853 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" podStartSLOduration=1.561824908 podStartE2EDuration="1.561824908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:18:25.560705509 +0000 UTC m=+1471.209324700" watchObservedRunningTime="2026-09-06 06:18:25.561824908 +0000 UTC m=+1471.210444129" Sep 06 06:18:35.023374 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:35.023336 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:35.023935 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:35.023390 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:35.035880 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:35.035857 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:18:35.611442 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:18:35.611410 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:19:08.362558 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.362519 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np"] Sep 06 06:19:08.365069 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.362761 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" podUID="b86441ec-02e4-408a-9ae9-f65ee084dc9f" containerName="payload-pre-processing" containerID="cri-o://74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde" gracePeriod=30 Sep 06 06:19:08.628495 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.628437 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" Sep 06 06:19:08.722784 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.722752 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z94bj\" (UniqueName: \"kubernetes.io/projected/b86441ec-02e4-408a-9ae9-f65ee084dc9f-kube-api-access-z94bj\") pod \"b86441ec-02e4-408a-9ae9-f65ee084dc9f\" (UID: \"b86441ec-02e4-408a-9ae9-f65ee084dc9f\") " Sep 06 06:19:08.722974 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.722917 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86441ec-02e4-408a-9ae9-f65ee084dc9f-plugins-config-volume\") pod \"b86441ec-02e4-408a-9ae9-f65ee084dc9f\" (UID: \"b86441ec-02e4-408a-9ae9-f65ee084dc9f\") " Sep 06 06:19:08.723257 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.723235 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86441ec-02e4-408a-9ae9-f65ee084dc9f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b86441ec-02e4-408a-9ae9-f65ee084dc9f" (UID: "b86441ec-02e4-408a-9ae9-f65ee084dc9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:19:08.724698 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.724669 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86441ec-02e4-408a-9ae9-f65ee084dc9f-kube-api-access-z94bj" (OuterVolumeSpecName: "kube-api-access-z94bj") pod "b86441ec-02e4-408a-9ae9-f65ee084dc9f" (UID: "b86441ec-02e4-408a-9ae9-f65ee084dc9f"). InnerVolumeSpecName "kube-api-access-z94bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:19:08.775848 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.775794 2580 generic.go:356] "Generic (PLEG): container finished" podID="b86441ec-02e4-408a-9ae9-f65ee084dc9f" containerID="74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde" exitCode=0 Sep 06 06:19:08.775963 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.775862 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" event={"ID":"b86441ec-02e4-408a-9ae9-f65ee084dc9f","Type":"ContainerDied","Data":"74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde"} Sep 06 06:19:08.775963 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.775941 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" event={"ID":"b86441ec-02e4-408a-9ae9-f65ee084dc9f","Type":"ContainerDied","Data":"f12dbcbc467b5ea3bbe1c82e16cba313292a4d3ebace157b91bb15734da4955b"} Sep 06 06:19:08.775963 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.775872 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np" Sep 06 06:19:08.775963 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.775963 2580 scope.go:117] "RemoveContainer" containerID="74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde" Sep 06 06:19:08.785060 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.785046 2580 scope.go:117] "RemoveContainer" containerID="74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde" Sep 06 06:19:08.785321 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:19:08.785305 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde\": container with ID starting with 74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde not found: ID does not exist" containerID="74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde" Sep 06 06:19:08.785376 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.785327 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde"} err="failed to get container status \"74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde\": rpc error: code = NotFound desc = could not find container \"74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde\": container with ID starting with 74fc6375f34ea17a0faf960ebf470110373f57dc2faebc52e7267361754c4bde not found: ID does not exist" Sep 06 06:19:08.797339 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.797310 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np"] Sep 06 06:19:08.800967 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.800949 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd8c231a-5f77bdf5db-k46np"] Sep 06 06:19:08.824119 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.824100 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z94bj\" (UniqueName: \"kubernetes.io/projected/b86441ec-02e4-408a-9ae9-f65ee084dc9f-kube-api-access-z94bj\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:19:08.824189 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.824120 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86441ec-02e4-408a-9ae9-f65ee084dc9f-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:19:08.875670 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:08.875648 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b86441ec-02e4-408a-9ae9-f65ee084dc9f" path="/var/lib/kubelet/pods/b86441ec-02e4-408a-9ae9-f65ee084dc9f/volumes" Sep 06 06:19:16.526996 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:16.526961 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g"] Sep 06 06:19:31.546225 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.546151 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" podUID="6cd62a68-f0f9-48bf-ac4f-2f6850572a84" containerName="main" containerID="cri-o://3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1" gracePeriod=45 Sep 06 06:19:31.800604 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.800552 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:19:31.898633 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.898598 2580 generic.go:356] "Generic (PLEG): container finished" podID="6cd62a68-f0f9-48bf-ac4f-2f6850572a84" containerID="3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1" exitCode=0 Sep 06 06:19:31.898633 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.898634 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" Sep 06 06:19:31.898905 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.898673 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" event={"ID":"6cd62a68-f0f9-48bf-ac4f-2f6850572a84","Type":"ContainerDied","Data":"3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1"} Sep 06 06:19:31.898905 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.898710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g" event={"ID":"6cd62a68-f0f9-48bf-ac4f-2f6850572a84","Type":"ContainerDied","Data":"2e6f64cf8f90eab0d28fccd6802104df9de3336e06f964525bc17535cbf3d6b7"} Sep 06 06:19:31.898905 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.898732 2580 scope.go:117] "RemoveContainer" containerID="3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1" Sep 06 06:19:31.907039 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.907019 2580 scope.go:117] "RemoveContainer" containerID="3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1" Sep 06 06:19:31.907289 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:19:31.907270 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1\": container with ID starting with 3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1 not found: ID does not exist" containerID="3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1" Sep 06 06:19:31.907352 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.907297 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1"} err="failed to get container status \"3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1\": rpc error: code = NotFound desc = could not find container \"3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1\": container with ID starting with 3c9501777cf5f436c74f93a55c36f57c2629411c62b900cdec2caa799c403aa1 not found: ID does not exist" Sep 06 06:19:31.929616 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.929594 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-tls-certs\") pod \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " Sep 06 06:19:31.929688 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.929625 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkr58\" (UniqueName: \"kubernetes.io/projected/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-kube-api-access-pkr58\") pod \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " Sep 06 06:19:31.929688 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.929663 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-tmp-dir\") pod \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " Sep 06 06:19:31.929797 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.929688 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-model-cache\") pod \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " Sep 06 06:19:31.929797 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.929771 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-home\") pod \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " Sep 06 06:19:31.929936 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.929815 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-dshm\") pod \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\" (UID: \"6cd62a68-f0f9-48bf-ac4f-2f6850572a84\") " Sep 06 06:19:31.930001 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.929979 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cd62a68-f0f9-48bf-ac4f-2f6850572a84" (UID: "6cd62a68-f0f9-48bf-ac4f-2f6850572a84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:19:31.930069 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.930015 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cd62a68-f0f9-48bf-ac4f-2f6850572a84" (UID: "6cd62a68-f0f9-48bf-ac4f-2f6850572a84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:19:31.930119 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.930071 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-home" (OuterVolumeSpecName: "home") pod "6cd62a68-f0f9-48bf-ac4f-2f6850572a84" (UID: "6cd62a68-f0f9-48bf-ac4f-2f6850572a84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:19:31.930377 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.930298 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-home\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:19:31.930377 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.930341 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-tmp-dir\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:19:31.930377 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.930351 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-model-cache\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:19:31.931899 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.931879 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cd62a68-f0f9-48bf-ac4f-2f6850572a84" (UID: "6cd62a68-f0f9-48bf-ac4f-2f6850572a84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:19:31.931979 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.931963 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-kube-api-access-pkr58" (OuterVolumeSpecName: "kube-api-access-pkr58") pod "6cd62a68-f0f9-48bf-ac4f-2f6850572a84" (UID: "6cd62a68-f0f9-48bf-ac4f-2f6850572a84"). InnerVolumeSpecName "kube-api-access-pkr58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:19:31.932021 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:31.931963 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-dshm" (OuterVolumeSpecName: "dshm") pod "6cd62a68-f0f9-48bf-ac4f-2f6850572a84" (UID: "6cd62a68-f0f9-48bf-ac4f-2f6850572a84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:19:32.030759 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:32.030736 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-tls-certs\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:19:32.030759 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:32.030759 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkr58\" (UniqueName: \"kubernetes.io/projected/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-kube-api-access-pkr58\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:19:32.030923 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:32.030768 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd62a68-f0f9-48bf-ac4f-2f6850572a84-dshm\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:19:32.224983 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:32.224952 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g"] Sep 06 06:19:32.228386 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:32.228361 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-dd8c231a/e2e-lifecycle-model-dd8c231a-kserve-7ccfd4bcd7-tlh7g"] Sep 06 06:19:32.880775 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:32.880741 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd62a68-f0f9-48bf-ac4f-2f6850572a84" path="/var/lib/kubelet/pods/6cd62a68-f0f9-48bf-ac4f-2f6850572a84/volumes" Sep 06 06:19:58.599146 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:19:58.599113 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd"] Sep 06 06:20:13.618869 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.618793 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" podUID="5cb20345-c78d-490b-9a29-4e1433e5f190" containerName="main" containerID="cri-o://3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630" gracePeriod=45 Sep 06 06:20:13.862641 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.862618 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:20:13.993913 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.993880 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-dshm\") pod \"5cb20345-c78d-490b-9a29-4e1433e5f190\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " Sep 06 06:20:13.994071 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.993964 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjm4k\" (UniqueName: \"kubernetes.io/projected/5cb20345-c78d-490b-9a29-4e1433e5f190-kube-api-access-zjm4k\") pod \"5cb20345-c78d-490b-9a29-4e1433e5f190\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " Sep 06 06:20:13.994071 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.994008 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb20345-c78d-490b-9a29-4e1433e5f190-tls-certs\") pod \"5cb20345-c78d-490b-9a29-4e1433e5f190\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " Sep 06 06:20:13.994071 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.994025 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-tmp-dir\") pod \"5cb20345-c78d-490b-9a29-4e1433e5f190\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " Sep 06 06:20:13.994071 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.994050 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-model-cache\") pod \"5cb20345-c78d-490b-9a29-4e1433e5f190\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " Sep 06 06:20:13.994295 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.994087 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-home\") pod \"5cb20345-c78d-490b-9a29-4e1433e5f190\" (UID: \"5cb20345-c78d-490b-9a29-4e1433e5f190\") " Sep 06 06:20:13.994365 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.994304 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5cb20345-c78d-490b-9a29-4e1433e5f190" (UID: "5cb20345-c78d-490b-9a29-4e1433e5f190"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:13.994424 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.994370 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-model-cache" (OuterVolumeSpecName: "model-cache") pod "5cb20345-c78d-490b-9a29-4e1433e5f190" (UID: "5cb20345-c78d-490b-9a29-4e1433e5f190"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:13.994465 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.994425 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-home" (OuterVolumeSpecName: "home") pod "5cb20345-c78d-490b-9a29-4e1433e5f190" (UID: "5cb20345-c78d-490b-9a29-4e1433e5f190"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:13.994465 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.994436 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-tmp-dir\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:20:13.996238 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.996213 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb20345-c78d-490b-9a29-4e1433e5f190-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5cb20345-c78d-490b-9a29-4e1433e5f190" (UID: "5cb20345-c78d-490b-9a29-4e1433e5f190"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:20:13.996339 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.996256 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb20345-c78d-490b-9a29-4e1433e5f190-kube-api-access-zjm4k" (OuterVolumeSpecName: "kube-api-access-zjm4k") pod "5cb20345-c78d-490b-9a29-4e1433e5f190" (UID: "5cb20345-c78d-490b-9a29-4e1433e5f190"). InnerVolumeSpecName "kube-api-access-zjm4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:20:13.996339 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:13.996305 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-dshm" (OuterVolumeSpecName: "dshm") pod "5cb20345-c78d-490b-9a29-4e1433e5f190" (UID: "5cb20345-c78d-490b-9a29-4e1433e5f190"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:14.095803 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.095774 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjm4k\" (UniqueName: \"kubernetes.io/projected/5cb20345-c78d-490b-9a29-4e1433e5f190-kube-api-access-zjm4k\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:20:14.095803 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.095798 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb20345-c78d-490b-9a29-4e1433e5f190-tls-certs\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:20:14.095966 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.095812 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-model-cache\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:20:14.095966 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.095823 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-home\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:20:14.095966 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.095847 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb20345-c78d-490b-9a29-4e1433e5f190-dshm\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:20:14.122686 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.122659 2580 generic.go:356] "Generic (PLEG): container finished" podID="5cb20345-c78d-490b-9a29-4e1433e5f190" containerID="3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630" exitCode=0 Sep 06 06:20:14.122796 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.122703 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" Sep 06 06:20:14.122796 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.122734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" event={"ID":"5cb20345-c78d-490b-9a29-4e1433e5f190","Type":"ContainerDied","Data":"3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630"} Sep 06 06:20:14.122796 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.122775 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd" event={"ID":"5cb20345-c78d-490b-9a29-4e1433e5f190","Type":"ContainerDied","Data":"b4c54aba571c5d9b213d3acca220e4580f6bc58473e8c356c544da1d1e8d3351"} Sep 06 06:20:14.122991 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.122797 2580 scope.go:117] "RemoveContainer" containerID="3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630" Sep 06 06:20:14.131847 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.131812 2580 scope.go:117] "RemoveContainer" containerID="3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630" Sep 06 06:20:14.132087 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:20:14.132068 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630\": container with ID starting with 3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630 not found: ID does not exist" containerID="3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630" Sep 06 06:20:14.132137 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.132094 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630"} err="failed to get container status \"3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630\": rpc error: code = NotFound desc = could not find container \"3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630\": container with ID starting with 3911101c2a962d65020466d5b76c7da1ac8ed22032bf291cac38bb0b19adf630 not found: ID does not exist" Sep 06 06:20:14.142935 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.142915 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd"] Sep 06 06:20:14.145655 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.145635 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-429831/auth-test-model-429831-kserve-566b5fb6ff-7tdjd"] Sep 06 06:20:14.878010 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:14.877966 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cb20345-c78d-490b-9a29-4e1433e5f190" path="/var/lib/kubelet/pods/5cb20345-c78d-490b-9a29-4e1433e5f190/volumes" Sep 06 06:20:58.211706 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.211675 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6"] Sep 06 06:20:58.212189 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.212174 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd62a68-f0f9-48bf-ac4f-2f6850572a84" containerName="main" Sep 06 06:20:58.212240 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.212192 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd62a68-f0f9-48bf-ac4f-2f6850572a84" containerName="main" Sep 06 06:20:58.212240 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.212204 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86441ec-02e4-408a-9ae9-f65ee084dc9f" containerName="payload-pre-processing" Sep 06 06:20:58.212240 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.212213 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86441ec-02e4-408a-9ae9-f65ee084dc9f" containerName="payload-pre-processing" Sep 06 06:20:58.212240 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.212231 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cb20345-c78d-490b-9a29-4e1433e5f190" containerName="main" Sep 06 06:20:58.212240 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.212236 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cb20345-c78d-490b-9a29-4e1433e5f190" containerName="main" Sep 06 06:20:58.212390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.212302 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd62a68-f0f9-48bf-ac4f-2f6850572a84" containerName="main" Sep 06 06:20:58.212390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.212315 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cb20345-c78d-490b-9a29-4e1433e5f190" containerName="main" Sep 06 06:20:58.212390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.212321 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b86441ec-02e4-408a-9ae9-f65ee084dc9f" containerName="payload-pre-processing" Sep 06 06:20:58.215382 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.215364 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" Sep 06 06:20:58.218195 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.218173 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ed85e8-dockercfg-wsbg2\"" Sep 06 06:20:58.218290 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.218206 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ed85e8\"" Sep 06 06:20:58.227344 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.227322 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6"] Sep 06 06:20:58.257214 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.257191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9cc7fc-2a60-4278-a603-d0762446a337-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6\" (UID: \"7e9cc7fc-2a60-4278-a603-d0762446a337\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" Sep 06 06:20:58.257310 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.257233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs928\" (UniqueName: \"kubernetes.io/projected/7e9cc7fc-2a60-4278-a603-d0762446a337-kube-api-access-gs928\") pod \"payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6\" (UID: \"7e9cc7fc-2a60-4278-a603-d0762446a337\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" Sep 06 06:20:58.357813 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.357780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9cc7fc-2a60-4278-a603-d0762446a337-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6\" (UID: \"7e9cc7fc-2a60-4278-a603-d0762446a337\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" Sep 06 06:20:58.357966 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.357838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs928\" (UniqueName: \"kubernetes.io/projected/7e9cc7fc-2a60-4278-a603-d0762446a337-kube-api-access-gs928\") pod \"payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6\" (UID: \"7e9cc7fc-2a60-4278-a603-d0762446a337\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" Sep 06 06:20:58.358443 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.358424 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9cc7fc-2a60-4278-a603-d0762446a337-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6\" (UID: \"7e9cc7fc-2a60-4278-a603-d0762446a337\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" Sep 06 06:20:58.365233 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.365210 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs928\" (UniqueName: \"kubernetes.io/projected/7e9cc7fc-2a60-4278-a603-d0762446a337-kube-api-access-gs928\") pod \"payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6\" (UID: \"7e9cc7fc-2a60-4278-a603-d0762446a337\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" Sep 06 06:20:58.527030 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.526936 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" Sep 06 06:20:58.800116 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:58.800086 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6"] Sep 06 06:20:58.801569 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:20:58.801542 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9cc7fc_2a60_4278_a603_d0762446a337.slice/crio-e4c7b8097329e20c3c0fb455b5dd503d74113635917c60cefa5941588bb79739 WatchSource:0}: Error finding container e4c7b8097329e20c3c0fb455b5dd503d74113635917c60cefa5941588bb79739: Status 404 returned error can't find the container with id e4c7b8097329e20c3c0fb455b5dd503d74113635917c60cefa5941588bb79739 Sep 06 06:20:59.357766 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:59.357686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" event={"ID":"7e9cc7fc-2a60-4278-a603-d0762446a337","Type":"ContainerStarted","Data":"721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f"} Sep 06 06:20:59.358333 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:59.357788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" event={"ID":"7e9cc7fc-2a60-4278-a603-d0762446a337","Type":"ContainerStarted","Data":"e4c7b8097329e20c3c0fb455b5dd503d74113635917c60cefa5941588bb79739"} Sep 06 06:20:59.358333 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:59.357841 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" Sep 06 06:20:59.378124 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:20:59.378074 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" podStartSLOduration=1.378059224 podStartE2EDuration="1.378059224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:20:59.375987027 +0000 UTC m=+1625.024606205" watchObservedRunningTime="2026-09-06 06:20:59.378059224 +0000 UTC m=+1625.026678402" Sep 06 06:21:10.364974 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:10.364944 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" Sep 06 06:21:14.010151 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.010120 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv"] Sep 06 06:21:14.018850 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.018800 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.021506 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.021477 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dc97f9\"/\"kube-root-ca.crt\"" Sep 06 06:21:14.021652 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.021520 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dc97f9\"/\"default-dockercfg-wj7sx\"" Sep 06 06:21:14.022460 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.022435 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv"] Sep 06 06:21:14.022703 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.022685 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dc97f9\"/\"sub-test-model-dc97f9-kserve-self-signed-certs\"" Sep 06 06:21:14.022792 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.022740 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dc97f9\"/\"openshift-service-ca.crt\"" Sep 06 06:21:14.104124 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.104098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-dshm\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.104250 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.104139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-tmp-dir\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.104250 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.104231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2fd70c2-2b91-4118-8025-d3c8795346bc-tls-certs\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.104331 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.104264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pdbv\" (UniqueName: \"kubernetes.io/projected/a2fd70c2-2b91-4118-8025-d3c8795346bc-kube-api-access-4pdbv\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.104331 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.104290 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-model-cache\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.104331 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.104305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-home\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.205594 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.205566 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2fd70c2-2b91-4118-8025-d3c8795346bc-tls-certs\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.205695 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.205600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdbv\" (UniqueName: \"kubernetes.io/projected/a2fd70c2-2b91-4118-8025-d3c8795346bc-kube-api-access-4pdbv\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.205695 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.205626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-model-cache\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.205695 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.205651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-home\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.205695 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.205684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-dshm\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.205853 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.205719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-tmp-dir\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.205984 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.205963 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-home\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.206060 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.206044 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-tmp-dir\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.206218 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.206199 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-model-cache\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.208021 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.208003 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-dshm\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.208214 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.208197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2fd70c2-2b91-4118-8025-d3c8795346bc-tls-certs\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.212982 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.212961 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pdbv\" (UniqueName: \"kubernetes.io/projected/a2fd70c2-2b91-4118-8025-d3c8795346bc-kube-api-access-4pdbv\") pod \"sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.330000 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.329940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:14.600536 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:14.600507 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv"] Sep 06 06:21:14.602636 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:21:14.602609 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2fd70c2_2b91_4118_8025_d3c8795346bc.slice/crio-72ec20c706fcba096c859a709d73d447e36e893f4ec20e4d4653356b85fc213a WatchSource:0}: Error finding container 72ec20c706fcba096c859a709d73d447e36e893f4ec20e4d4653356b85fc213a: Status 404 returned error can't find the container with id 72ec20c706fcba096c859a709d73d447e36e893f4ec20e4d4653356b85fc213a Sep 06 06:21:15.446710 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:15.446677 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" event={"ID":"a2fd70c2-2b91-4118-8025-d3c8795346bc","Type":"ContainerStarted","Data":"fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0"} Sep 06 06:21:15.446710 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:15.446718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" event={"ID":"a2fd70c2-2b91-4118-8025-d3c8795346bc","Type":"ContainerStarted","Data":"72ec20c706fcba096c859a709d73d447e36e893f4ec20e4d4653356b85fc213a"} Sep 06 06:21:15.463102 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:15.463055 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" podStartSLOduration=2.463040161 podStartE2EDuration="2.463040161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:21:15.461818887 +0000 UTC m=+1641.110438064" watchObservedRunningTime="2026-09-06 06:21:15.463040161 +0000 UTC m=+1641.111659384" Sep 06 06:21:24.330764 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:24.330733 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:24.331221 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:24.330795 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:24.342969 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:24.342944 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:24.505297 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:24.505272 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:21:27.329071 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.329033 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp"] Sep 06 06:21:27.332750 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.332730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" Sep 06 06:21:27.336433 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.336415 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1c55e7-dockercfg-v5gdm\"" Sep 06 06:21:27.336540 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.336519 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1c55e7\"" Sep 06 06:21:27.343412 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.343391 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp"] Sep 06 06:21:27.429565 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.429532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp\" (UID: \"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" Sep 06 06:21:27.429713 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.429615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ppz7\" (UniqueName: \"kubernetes.io/projected/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2-kube-api-access-2ppz7\") pod \"payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp\" (UID: \"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" Sep 06 06:21:27.531096 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.531063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ppz7\" (UniqueName: \"kubernetes.io/projected/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2-kube-api-access-2ppz7\") pod \"payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp\" (UID: \"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" Sep 06 06:21:27.531259 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.531164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp\" (UID: \"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" Sep 06 06:21:27.531812 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.531789 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp\" (UID: \"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" Sep 06 06:21:27.538783 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.538759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ppz7\" (UniqueName: \"kubernetes.io/projected/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2-kube-api-access-2ppz7\") pod \"payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp\" (UID: \"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" Sep 06 06:21:27.644596 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.644528 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" Sep 06 06:21:27.915046 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:27.915012 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp"] Sep 06 06:21:27.916503 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:21:27.916475 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0559bf8_75a5_430e_8ef2_cd5b78e33ed2.slice/crio-410912e4265e0e496af31d39153aac0fd2486ec3769e42eb6555c6b685006c9c WatchSource:0}: Error finding container 410912e4265e0e496af31d39153aac0fd2486ec3769e42eb6555c6b685006c9c: Status 404 returned error can't find the container with id 410912e4265e0e496af31d39153aac0fd2486ec3769e42eb6555c6b685006c9c Sep 06 06:21:28.518274 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:28.518235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" event={"ID":"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2","Type":"ContainerStarted","Data":"c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a"} Sep 06 06:21:28.518685 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:28.518304 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" event={"ID":"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2","Type":"ContainerStarted","Data":"410912e4265e0e496af31d39153aac0fd2486ec3769e42eb6555c6b685006c9c"} Sep 06 06:21:28.518685 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:28.518351 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" Sep 06 06:21:39.525040 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:39.524994 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" Sep 06 06:21:39.545354 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:39.545296 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" podStartSLOduration=12.545281253 podStartE2EDuration="12.545281253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:21:28.533323721 +0000 UTC m=+1654.181942914" watchObservedRunningTime="2026-09-06 06:21:39.545281253 +0000 UTC m=+1665.193900477" Sep 06 06:21:53.277861 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:53.277817 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4"] Sep 06 06:21:53.282785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:53.282768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4" Sep 06 06:21:53.285595 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:53.285558 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:21:53.285715 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:53.285681 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:21:53.287447 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:53.287421 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4"] Sep 06 06:21:53.358957 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:53.358927 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36472a3-9030-4eba-b4e8-f38a3e615792-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4\" (UID: \"b36472a3-9030-4eba-b4e8-f38a3e615792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4" Sep 06 06:21:53.460186 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:53.460150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36472a3-9030-4eba-b4e8-f38a3e615792-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4\" (UID: \"b36472a3-9030-4eba-b4e8-f38a3e615792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4" Sep 06 06:21:53.460729 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:53.460711 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36472a3-9030-4eba-b4e8-f38a3e615792-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4\" (UID: \"b36472a3-9030-4eba-b4e8-f38a3e615792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4" Sep 06 06:21:53.596212 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:53.596134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4" Sep 06 06:21:53.864265 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:53.864234 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4"] Sep 06 06:21:53.866169 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:21:53.866142 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36472a3_9030_4eba_b4e8_f38a3e615792.slice/crio-cf8ad38cf9f3e0f020f944691497099a3ccee3b2444efc6787535bdbfba958a2 WatchSource:0}: Error finding container cf8ad38cf9f3e0f020f944691497099a3ccee3b2444efc6787535bdbfba958a2: Status 404 returned error can't find the container with id cf8ad38cf9f3e0f020f944691497099a3ccee3b2444efc6787535bdbfba958a2 Sep 06 06:21:54.669434 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:54.669405 2580 generic.go:356] "Generic (PLEG): container finished" podID="b36472a3-9030-4eba-b4e8-f38a3e615792" containerID="432fd128ae2b1574f86540b250c8dc2bef03e5f51f0b20a21f3dfca45806e17f" exitCode=0 Sep 06 06:21:54.669785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:54.669496 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4" event={"ID":"b36472a3-9030-4eba-b4e8-f38a3e615792","Type":"ContainerDied","Data":"432fd128ae2b1574f86540b250c8dc2bef03e5f51f0b20a21f3dfca45806e17f"} Sep 06 06:21:54.669785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:54.669547 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4" event={"ID":"b36472a3-9030-4eba-b4e8-f38a3e615792","Type":"ContainerStarted","Data":"cf8ad38cf9f3e0f020f944691497099a3ccee3b2444efc6787535bdbfba958a2"} Sep 06 06:21:55.031977 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:55.031951 2580 scope.go:117] "RemoveContainer" containerID="baff5a6b5b33bfaf81fb98a6b1564304d41d5c2ce9a5029aa9201655bdc750d6" Sep 06 06:21:55.810496 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:55.810470 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4" Sep 06 06:21:55.882035 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:55.882000 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36472a3-9030-4eba-b4e8-f38a3e615792-maas-api-service-ca\") pod \"b36472a3-9030-4eba-b4e8-f38a3e615792\" (UID: \"b36472a3-9030-4eba-b4e8-f38a3e615792\") " Sep 06 06:21:55.882391 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:55.882370 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36472a3-9030-4eba-b4e8-f38a3e615792-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b36472a3-9030-4eba-b4e8-f38a3e615792" (UID: "b36472a3-9030-4eba-b4e8-f38a3e615792"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:21:55.983696 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:55.983616 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36472a3-9030-4eba-b4e8-f38a3e615792-maas-api-service-ca\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:21:56.682137 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:56.682106 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4" Sep 06 06:21:56.682327 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:56.682137 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4" event={"ID":"b36472a3-9030-4eba-b4e8-f38a3e615792","Type":"ContainerDied","Data":"cf8ad38cf9f3e0f020f944691497099a3ccee3b2444efc6787535bdbfba958a2"} Sep 06 06:21:56.682327 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:21:56.682184 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf8ad38cf9f3e0f020f944691497099a3ccee3b2444efc6787535bdbfba958a2" Sep 06 06:22:09.128526 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.128494 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6"] Sep 06 06:22:09.129319 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.128695 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" podUID="7e9cc7fc-2a60-4278-a603-d0762446a337" containerName="payload-pre-processing" containerID="cri-o://721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f" gracePeriod=30 Sep 06 06:22:09.275846 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.275797 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4"] Sep 06 06:22:09.277817 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.277788 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed85e8-t27z4"] Sep 06 06:22:09.381413 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.381360 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" Sep 06 06:22:09.498639 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.498605 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9cc7fc-2a60-4278-a603-d0762446a337-plugins-config-volume\") pod \"7e9cc7fc-2a60-4278-a603-d0762446a337\" (UID: \"7e9cc7fc-2a60-4278-a603-d0762446a337\") " Sep 06 06:22:09.498791 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.498679 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs928\" (UniqueName: \"kubernetes.io/projected/7e9cc7fc-2a60-4278-a603-d0762446a337-kube-api-access-gs928\") pod \"7e9cc7fc-2a60-4278-a603-d0762446a337\" (UID: \"7e9cc7fc-2a60-4278-a603-d0762446a337\") " Sep 06 06:22:09.499077 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.499049 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9cc7fc-2a60-4278-a603-d0762446a337-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e9cc7fc-2a60-4278-a603-d0762446a337" (UID: "7e9cc7fc-2a60-4278-a603-d0762446a337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:22:09.500715 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.500692 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9cc7fc-2a60-4278-a603-d0762446a337-kube-api-access-gs928" (OuterVolumeSpecName: "kube-api-access-gs928") pod "7e9cc7fc-2a60-4278-a603-d0762446a337" (UID: "7e9cc7fc-2a60-4278-a603-d0762446a337"). InnerVolumeSpecName "kube-api-access-gs928". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:09.599882 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.599848 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs928\" (UniqueName: \"kubernetes.io/projected/7e9cc7fc-2a60-4278-a603-d0762446a337-kube-api-access-gs928\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:09.599882 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.599876 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9cc7fc-2a60-4278-a603-d0762446a337-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:09.753137 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.753093 2580 generic.go:356] "Generic (PLEG): container finished" podID="7e9cc7fc-2a60-4278-a603-d0762446a337" containerID="721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f" exitCode=0 Sep 06 06:22:09.753281 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.753140 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" Sep 06 06:22:09.753281 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.753171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" event={"ID":"7e9cc7fc-2a60-4278-a603-d0762446a337","Type":"ContainerDied","Data":"721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f"} Sep 06 06:22:09.753281 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.753241 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6" event={"ID":"7e9cc7fc-2a60-4278-a603-d0762446a337","Type":"ContainerDied","Data":"e4c7b8097329e20c3c0fb455b5dd503d74113635917c60cefa5941588bb79739"} Sep 06 06:22:09.753281 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.753254 2580 scope.go:117] "RemoveContainer" containerID="721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f" Sep 06 06:22:09.764731 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.764697 2580 scope.go:117] "RemoveContainer" containerID="721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f" Sep 06 06:22:09.765133 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:22:09.765064 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f\": container with ID starting with 721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f not found: ID does not exist" containerID="721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f" Sep 06 06:22:09.765133 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.765099 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f"} err="failed to get container status \"721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f\": rpc error: code = NotFound desc = could not find container \"721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f\": container with ID starting with 721a3d56187ea6cd35c01e8e6063243b62e667dbf9956df95acfc2f015bd6b4f not found: ID does not exist" Sep 06 06:22:09.776873 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.776850 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6"] Sep 06 06:22:09.778770 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:09.778751 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ed85e8-6c7c5d9958-r56s6"] Sep 06 06:22:10.875366 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:10.875332 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e9cc7fc-2a60-4278-a603-d0762446a337" path="/var/lib/kubelet/pods/7e9cc7fc-2a60-4278-a603-d0762446a337/volumes" Sep 06 06:22:10.875781 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:10.875763 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b36472a3-9030-4eba-b4e8-f38a3e615792" path="/var/lib/kubelet/pods/b36472a3-9030-4eba-b4e8-f38a3e615792/volumes" Sep 06 06:22:16.302760 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.302726 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn"] Sep 06 06:22:16.303206 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.303192 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9cc7fc-2a60-4278-a603-d0762446a337" containerName="payload-pre-processing" Sep 06 06:22:16.303263 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.303207 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9cc7fc-2a60-4278-a603-d0762446a337" containerName="payload-pre-processing" Sep 06 06:22:16.303263 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.303228 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36472a3-9030-4eba-b4e8-f38a3e615792" containerName="revoke-keys" Sep 06 06:22:16.303263 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.303234 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36472a3-9030-4eba-b4e8-f38a3e615792" containerName="revoke-keys" Sep 06 06:22:16.303374 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.303308 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9cc7fc-2a60-4278-a603-d0762446a337" containerName="payload-pre-processing" Sep 06 06:22:16.303374 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.303318 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36472a3-9030-4eba-b4e8-f38a3e615792" containerName="revoke-keys" Sep 06 06:22:16.307961 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.307941 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn" Sep 06 06:22:16.311545 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.311524 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:22:16.311859 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.311581 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:22:16.312962 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.312942 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn"] Sep 06 06:22:16.463041 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.463012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfe8a6-20d3-4b66-a547-703398fa798d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn\" (UID: \"6ddfe8a6-20d3-4b66-a547-703398fa798d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn" Sep 06 06:22:16.564347 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.564278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfe8a6-20d3-4b66-a547-703398fa798d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn\" (UID: \"6ddfe8a6-20d3-4b66-a547-703398fa798d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn" Sep 06 06:22:16.564883 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.564863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfe8a6-20d3-4b66-a547-703398fa798d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn\" (UID: \"6ddfe8a6-20d3-4b66-a547-703398fa798d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn" Sep 06 06:22:16.619412 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.619390 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn" Sep 06 06:22:16.889540 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:16.889518 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn"] Sep 06 06:22:16.890773 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:22:16.890747 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddfe8a6_20d3_4b66_a547_703398fa798d.slice/crio-d5a54618313fd14ead8f9f42a8feb0c581d4e1214fbf047629518c44ced35a6e WatchSource:0}: Error finding container d5a54618313fd14ead8f9f42a8feb0c581d4e1214fbf047629518c44ced35a6e: Status 404 returned error can't find the container with id d5a54618313fd14ead8f9f42a8feb0c581d4e1214fbf047629518c44ced35a6e Sep 06 06:22:17.797866 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:17.797808 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ddfe8a6-20d3-4b66-a547-703398fa798d" containerID="1cce6554b9b68a3ee1626f85a41082870c76721d81b895c6d1d6ccdc6ebe4f38" exitCode=0 Sep 06 06:22:17.798201 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:17.797890 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn" event={"ID":"6ddfe8a6-20d3-4b66-a547-703398fa798d","Type":"ContainerDied","Data":"1cce6554b9b68a3ee1626f85a41082870c76721d81b895c6d1d6ccdc6ebe4f38"} Sep 06 06:22:17.798201 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:17.797942 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn" event={"ID":"6ddfe8a6-20d3-4b66-a547-703398fa798d","Type":"ContainerStarted","Data":"d5a54618313fd14ead8f9f42a8feb0c581d4e1214fbf047629518c44ced35a6e"} Sep 06 06:22:18.936678 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:18.936655 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn" Sep 06 06:22:19.085103 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:19.085031 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfe8a6-20d3-4b66-a547-703398fa798d-maas-api-service-ca\") pod \"6ddfe8a6-20d3-4b66-a547-703398fa798d\" (UID: \"6ddfe8a6-20d3-4b66-a547-703398fa798d\") " Sep 06 06:22:19.085381 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:19.085356 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddfe8a6-20d3-4b66-a547-703398fa798d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ddfe8a6-20d3-4b66-a547-703398fa798d" (UID: "6ddfe8a6-20d3-4b66-a547-703398fa798d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:22:19.185903 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:19.185875 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfe8a6-20d3-4b66-a547-703398fa798d-maas-api-service-ca\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:19.809765 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:19.809728 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn" event={"ID":"6ddfe8a6-20d3-4b66-a547-703398fa798d","Type":"ContainerDied","Data":"d5a54618313fd14ead8f9f42a8feb0c581d4e1214fbf047629518c44ced35a6e"} Sep 06 06:22:19.809765 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:19.809765 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn" Sep 06 06:22:19.809975 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:19.809801 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5a54618313fd14ead8f9f42a8feb0c581d4e1214fbf047629518c44ced35a6e" Sep 06 06:22:21.118587 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:21.118550 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv"] Sep 06 06:22:31.762070 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:31.762023 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp"] Sep 06 06:22:31.762584 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:31.762269 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" podUID="d0559bf8-75a5-430e-8ef2-cd5b78e33ed2" containerName="payload-processing" containerID="cri-o://c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a" gracePeriod=30 Sep 06 06:22:31.921232 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:31.921201 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn"] Sep 06 06:22:31.923482 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:31.923456 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c55e7-x4mnn"] Sep 06 06:22:32.015793 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.015733 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" Sep 06 06:22:32.098449 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.098418 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2-plugins-config-volume\") pod \"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2\" (UID: \"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2\") " Sep 06 06:22:32.098449 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.098449 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ppz7\" (UniqueName: \"kubernetes.io/projected/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2-kube-api-access-2ppz7\") pod \"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2\" (UID: \"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2\") " Sep 06 06:22:32.098774 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.098752 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0559bf8-75a5-430e-8ef2-cd5b78e33ed2" (UID: "d0559bf8-75a5-430e-8ef2-cd5b78e33ed2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:22:32.100550 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.100525 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2-kube-api-access-2ppz7" (OuterVolumeSpecName: "kube-api-access-2ppz7") pod "d0559bf8-75a5-430e-8ef2-cd5b78e33ed2" (UID: "d0559bf8-75a5-430e-8ef2-cd5b78e33ed2"). InnerVolumeSpecName "kube-api-access-2ppz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:32.205806 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.200257 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:32.205806 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.200296 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ppz7\" (UniqueName: \"kubernetes.io/projected/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2-kube-api-access-2ppz7\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:32.878662 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.878612 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ddfe8a6-20d3-4b66-a547-703398fa798d" path="/var/lib/kubelet/pods/6ddfe8a6-20d3-4b66-a547-703398fa798d/volumes" Sep 06 06:22:32.888522 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.888491 2580 generic.go:356] "Generic (PLEG): container finished" podID="d0559bf8-75a5-430e-8ef2-cd5b78e33ed2" containerID="c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a" exitCode=0 Sep 06 06:22:32.888651 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.888544 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" event={"ID":"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2","Type":"ContainerDied","Data":"c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a"} Sep 06 06:22:32.888651 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.888552 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" Sep 06 06:22:32.888651 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.888581 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp" event={"ID":"d0559bf8-75a5-430e-8ef2-cd5b78e33ed2","Type":"ContainerDied","Data":"410912e4265e0e496af31d39153aac0fd2486ec3769e42eb6555c6b685006c9c"} Sep 06 06:22:32.888651 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.888602 2580 scope.go:117] "RemoveContainer" containerID="c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a" Sep 06 06:22:32.897576 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.897557 2580 scope.go:117] "RemoveContainer" containerID="c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a" Sep 06 06:22:32.897865 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:22:32.897844 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a\": container with ID starting with c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a not found: ID does not exist" containerID="c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a" Sep 06 06:22:32.897910 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.897874 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a"} err="failed to get container status \"c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a\": rpc error: code = NotFound desc = could not find container \"c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a\": container with ID starting with c8ecf884187506af2901ff17e2c020a77571c3d66e6da7b3f9bfd68213ef410a not found: ID does not exist" Sep 06 06:22:32.905071 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.905049 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp"] Sep 06 06:22:32.911160 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:32.911137 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c55e7-77f6f8dcbf-7hzmp"] Sep 06 06:22:34.875612 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:34.875578 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0559bf8-75a5-430e-8ef2-cd5b78e33ed2" path="/var/lib/kubelet/pods/d0559bf8-75a5-430e-8ef2-cd5b78e33ed2/volumes" Sep 06 06:22:36.138141 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.138106 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" podUID="a2fd70c2-2b91-4118-8025-d3c8795346bc" containerName="main" containerID="cri-o://fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0" gracePeriod=45 Sep 06 06:22:36.393311 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.393245 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:22:36.438107 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.438076 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-model-cache\") pod \"a2fd70c2-2b91-4118-8025-d3c8795346bc\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " Sep 06 06:22:36.438259 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.438133 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-home\") pod \"a2fd70c2-2b91-4118-8025-d3c8795346bc\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " Sep 06 06:22:36.438259 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.438168 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pdbv\" (UniqueName: \"kubernetes.io/projected/a2fd70c2-2b91-4118-8025-d3c8795346bc-kube-api-access-4pdbv\") pod \"a2fd70c2-2b91-4118-8025-d3c8795346bc\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " Sep 06 06:22:36.438259 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.438205 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-tmp-dir\") pod \"a2fd70c2-2b91-4118-8025-d3c8795346bc\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " Sep 06 06:22:36.438259 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.438254 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2fd70c2-2b91-4118-8025-d3c8795346bc-tls-certs\") pod \"a2fd70c2-2b91-4118-8025-d3c8795346bc\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " Sep 06 06:22:36.438448 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.438293 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-dshm\") pod \"a2fd70c2-2b91-4118-8025-d3c8795346bc\" (UID: \"a2fd70c2-2b91-4118-8025-d3c8795346bc\") " Sep 06 06:22:36.438448 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.438426 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2fd70c2-2b91-4118-8025-d3c8795346bc" (UID: "a2fd70c2-2b91-4118-8025-d3c8795346bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:36.438560 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.438455 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2fd70c2-2b91-4118-8025-d3c8795346bc" (UID: "a2fd70c2-2b91-4118-8025-d3c8795346bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:36.438560 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.438477 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-home" (OuterVolumeSpecName: "home") pod "a2fd70c2-2b91-4118-8025-d3c8795346bc" (UID: "a2fd70c2-2b91-4118-8025-d3c8795346bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:36.438674 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.438627 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-model-cache\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:36.438674 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.438643 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-home\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:36.438674 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.438652 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-tmp-dir\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:36.440542 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.440507 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2fd70c2-2b91-4118-8025-d3c8795346bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2fd70c2-2b91-4118-8025-d3c8795346bc" (UID: "a2fd70c2-2b91-4118-8025-d3c8795346bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:22:36.440662 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.440593 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fd70c2-2b91-4118-8025-d3c8795346bc-kube-api-access-4pdbv" (OuterVolumeSpecName: "kube-api-access-4pdbv") pod "a2fd70c2-2b91-4118-8025-d3c8795346bc" (UID: "a2fd70c2-2b91-4118-8025-d3c8795346bc"). InnerVolumeSpecName "kube-api-access-4pdbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:36.440735 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.440681 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-dshm" (OuterVolumeSpecName: "dshm") pod "a2fd70c2-2b91-4118-8025-d3c8795346bc" (UID: "a2fd70c2-2b91-4118-8025-d3c8795346bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:36.539365 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.539337 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pdbv\" (UniqueName: \"kubernetes.io/projected/a2fd70c2-2b91-4118-8025-d3c8795346bc-kube-api-access-4pdbv\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:36.539365 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.539360 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2fd70c2-2b91-4118-8025-d3c8795346bc-tls-certs\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:36.539365 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.539371 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2fd70c2-2b91-4118-8025-d3c8795346bc-dshm\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:36.911836 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.911804 2580 generic.go:356] "Generic (PLEG): container finished" podID="a2fd70c2-2b91-4118-8025-d3c8795346bc" containerID="fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0" exitCode=0 Sep 06 06:22:36.911950 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.911874 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" Sep 06 06:22:36.911950 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.911886 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" event={"ID":"a2fd70c2-2b91-4118-8025-d3c8795346bc","Type":"ContainerDied","Data":"fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0"} Sep 06 06:22:36.911950 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.911927 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv" event={"ID":"a2fd70c2-2b91-4118-8025-d3c8795346bc","Type":"ContainerDied","Data":"72ec20c706fcba096c859a709d73d447e36e893f4ec20e4d4653356b85fc213a"} Sep 06 06:22:36.911950 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.911944 2580 scope.go:117] "RemoveContainer" containerID="fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0" Sep 06 06:22:36.920712 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.920682 2580 scope.go:117] "RemoveContainer" containerID="fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0" Sep 06 06:22:36.920993 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:22:36.920971 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0\": container with ID starting with fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0 not found: ID does not exist" containerID="fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0" Sep 06 06:22:36.921065 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.921003 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0"} err="failed to get container status \"fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0\": rpc error: code = NotFound desc = could not find container \"fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0\": container with ID starting with fd54225ae9b309c088be4fd02a6b9abf32a3dbcc14efb9f98d4fbb56c727eec0 not found: ID does not exist" Sep 06 06:22:36.931363 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.931331 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv"] Sep 06 06:22:36.935928 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:36.935906 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/sub-test-model-dc97f9-kserve-648bd9b44c-jt8bv"] Sep 06 06:22:38.875599 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:38.875567 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2fd70c2-2b91-4118-8025-d3c8795346bc" path="/var/lib/kubelet/pods/a2fd70c2-2b91-4118-8025-d3c8795346bc/volumes" Sep 06 06:22:49.628434 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.628396 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f"] Sep 06 06:22:49.629246 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.628652 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" podUID="30148597-c0f0-4d25-954f-2cd5db81152c" containerName="payload-processing" containerID="cri-o://7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb" gracePeriod=30 Sep 06 06:22:49.638590 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.638562 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f"] Sep 06 06:22:49.638814 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.638788 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" podUID="25372e71-56b0-4221-bb07-5d1fddc4fdda" containerName="payload-pre-processing" containerID="cri-o://d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b" gracePeriod=30 Sep 06 06:22:49.924398 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.924371 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" Sep 06 06:22:49.926685 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.926666 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" Sep 06 06:22:49.979709 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.979677 2580 generic.go:356] "Generic (PLEG): container finished" podID="30148597-c0f0-4d25-954f-2cd5db81152c" containerID="7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb" exitCode=0 Sep 06 06:22:49.979902 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.979717 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" Sep 06 06:22:49.979902 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.979737 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" event={"ID":"30148597-c0f0-4d25-954f-2cd5db81152c","Type":"ContainerDied","Data":"7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb"} Sep 06 06:22:49.979902 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.979851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f" event={"ID":"30148597-c0f0-4d25-954f-2cd5db81152c","Type":"ContainerDied","Data":"d385aacd9f1c6873a6e3dbaa29e7597430211274d1627c054cedeecf55267b2e"} Sep 06 06:22:49.979902 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.979876 2580 scope.go:117] "RemoveContainer" containerID="7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb" Sep 06 06:22:49.981650 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.981624 2580 generic.go:356] "Generic (PLEG): container finished" podID="25372e71-56b0-4221-bb07-5d1fddc4fdda" containerID="d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b" exitCode=0 Sep 06 06:22:49.981759 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.981677 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" Sep 06 06:22:49.981759 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.981690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" event={"ID":"25372e71-56b0-4221-bb07-5d1fddc4fdda","Type":"ContainerDied","Data":"d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b"} Sep 06 06:22:49.981759 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.981729 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f" event={"ID":"25372e71-56b0-4221-bb07-5d1fddc4fdda","Type":"ContainerDied","Data":"595c7dc551bd69884cf2274b20cd231720a31114340db4f9cc1cc2c777904954"} Sep 06 06:22:49.989453 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.989433 2580 scope.go:117] "RemoveContainer" containerID="7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb" Sep 06 06:22:49.989764 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:22:49.989739 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb\": container with ID starting with 7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb not found: ID does not exist" containerID="7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb" Sep 06 06:22:49.989870 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.989787 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb"} err="failed to get container status \"7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb\": rpc error: code = NotFound desc = could not find container \"7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb\": container with ID starting with 7e0985e2db032ac94b93cdb8f2b0c0122a6959276ab84e054dc9a7de5b3fe1bb not found: ID does not exist" Sep 06 06:22:49.989870 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.989812 2580 scope.go:117] "RemoveContainer" containerID="d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b" Sep 06 06:22:49.998306 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.998287 2580 scope.go:117] "RemoveContainer" containerID="d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b" Sep 06 06:22:49.998571 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:22:49.998547 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b\": container with ID starting with d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b not found: ID does not exist" containerID="d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b" Sep 06 06:22:49.998632 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:49.998577 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b"} err="failed to get container status \"d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b\": rpc error: code = NotFound desc = could not find container \"d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b\": container with ID starting with d61a8023fc26784a0bbf15b9ff79194e6a00d0add770adda92fe5ab0009fce5b not found: ID does not exist" Sep 06 06:22:50.067549 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.067518 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30148597-c0f0-4d25-954f-2cd5db81152c-plugins-config-volume\") pod \"30148597-c0f0-4d25-954f-2cd5db81152c\" (UID: \"30148597-c0f0-4d25-954f-2cd5db81152c\") " Sep 06 06:22:50.067549 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.067554 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25372e71-56b0-4221-bb07-5d1fddc4fdda-plugins-config-volume\") pod \"25372e71-56b0-4221-bb07-5d1fddc4fdda\" (UID: \"25372e71-56b0-4221-bb07-5d1fddc4fdda\") " Sep 06 06:22:50.067719 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.067586 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4rcn\" (UniqueName: \"kubernetes.io/projected/25372e71-56b0-4221-bb07-5d1fddc4fdda-kube-api-access-h4rcn\") pod \"25372e71-56b0-4221-bb07-5d1fddc4fdda\" (UID: \"25372e71-56b0-4221-bb07-5d1fddc4fdda\") " Sep 06 06:22:50.067719 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.067643 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x44d\" (UniqueName: \"kubernetes.io/projected/30148597-c0f0-4d25-954f-2cd5db81152c-kube-api-access-5x44d\") pod \"30148597-c0f0-4d25-954f-2cd5db81152c\" (UID: \"30148597-c0f0-4d25-954f-2cd5db81152c\") " Sep 06 06:22:50.067937 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.067913 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30148597-c0f0-4d25-954f-2cd5db81152c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "30148597-c0f0-4d25-954f-2cd5db81152c" (UID: "30148597-c0f0-4d25-954f-2cd5db81152c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:22:50.067999 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.067965 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25372e71-56b0-4221-bb07-5d1fddc4fdda-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25372e71-56b0-4221-bb07-5d1fddc4fdda" (UID: "25372e71-56b0-4221-bb07-5d1fddc4fdda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:22:50.068077 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.068064 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30148597-c0f0-4d25-954f-2cd5db81152c-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:50.068123 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.068080 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25372e71-56b0-4221-bb07-5d1fddc4fdda-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:50.069785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.069763 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30148597-c0f0-4d25-954f-2cd5db81152c-kube-api-access-5x44d" (OuterVolumeSpecName: "kube-api-access-5x44d") pod "30148597-c0f0-4d25-954f-2cd5db81152c" (UID: "30148597-c0f0-4d25-954f-2cd5db81152c"). InnerVolumeSpecName "kube-api-access-5x44d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:50.069849 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.069801 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25372e71-56b0-4221-bb07-5d1fddc4fdda-kube-api-access-h4rcn" (OuterVolumeSpecName: "kube-api-access-h4rcn") pod "25372e71-56b0-4221-bb07-5d1fddc4fdda" (UID: "25372e71-56b0-4221-bb07-5d1fddc4fdda"). InnerVolumeSpecName "kube-api-access-h4rcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:50.168820 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.168791 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4rcn\" (UniqueName: \"kubernetes.io/projected/25372e71-56b0-4221-bb07-5d1fddc4fdda-kube-api-access-h4rcn\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:50.168820 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.168816 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x44d\" (UniqueName: \"kubernetes.io/projected/30148597-c0f0-4d25-954f-2cd5db81152c-kube-api-access-5x44d\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:22:50.302092 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.302061 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f"] Sep 06 06:22:50.305301 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.305275 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-340e00-5bfc8d66fc-rbs2f"] Sep 06 06:22:50.318159 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.318120 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f"] Sep 06 06:22:50.323478 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.323451 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-340e00-f775d4646-v4q6f"] Sep 06 06:22:50.876175 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.876143 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25372e71-56b0-4221-bb07-5d1fddc4fdda" path="/var/lib/kubelet/pods/25372e71-56b0-4221-bb07-5d1fddc4fdda/volumes" Sep 06 06:22:50.876531 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:50.876501 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30148597-c0f0-4d25-954f-2cd5db81152c" path="/var/lib/kubelet/pods/30148597-c0f0-4d25-954f-2cd5db81152c/volumes" Sep 06 06:22:58.600770 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.600733 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk"] Sep 06 06:22:58.601472 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601447 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0559bf8-75a5-430e-8ef2-cd5b78e33ed2" containerName="payload-processing" Sep 06 06:22:58.601650 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601626 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0559bf8-75a5-430e-8ef2-cd5b78e33ed2" containerName="payload-processing" Sep 06 06:22:58.601785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601660 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30148597-c0f0-4d25-954f-2cd5db81152c" containerName="payload-processing" Sep 06 06:22:58.601785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601671 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="30148597-c0f0-4d25-954f-2cd5db81152c" containerName="payload-processing" Sep 06 06:22:58.601785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601694 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ddfe8a6-20d3-4b66-a547-703398fa798d" containerName="revoke-keys" Sep 06 06:22:58.601785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601703 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ddfe8a6-20d3-4b66-a547-703398fa798d" containerName="revoke-keys" Sep 06 06:22:58.601785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601734 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2fd70c2-2b91-4118-8025-d3c8795346bc" containerName="main" Sep 06 06:22:58.601785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601743 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2fd70c2-2b91-4118-8025-d3c8795346bc" containerName="main" Sep 06 06:22:58.601785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601761 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25372e71-56b0-4221-bb07-5d1fddc4fdda" containerName="payload-pre-processing" Sep 06 06:22:58.601785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601769 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="25372e71-56b0-4221-bb07-5d1fddc4fdda" containerName="payload-pre-processing" Sep 06 06:22:58.602246 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601899 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2fd70c2-2b91-4118-8025-d3c8795346bc" containerName="main" Sep 06 06:22:58.602246 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601913 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ddfe8a6-20d3-4b66-a547-703398fa798d" containerName="revoke-keys" Sep 06 06:22:58.602246 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601924 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="30148597-c0f0-4d25-954f-2cd5db81152c" containerName="payload-processing" Sep 06 06:22:58.602246 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601932 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="25372e71-56b0-4221-bb07-5d1fddc4fdda" containerName="payload-pre-processing" Sep 06 06:22:58.602246 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.601944 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0559bf8-75a5-430e-8ef2-cd5b78e33ed2" containerName="payload-processing" Sep 06 06:22:58.605433 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.605410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.609447 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.609421 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dc97f9\"/\"kube-root-ca.crt\"" Sep 06 06:22:58.609629 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.609487 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dc97f9\"/\"default-dockercfg-wj7sx\"" Sep 06 06:22:58.609629 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.609495 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dc97f9\"/\"rate-test-model-dc97f9-kserve-self-signed-certs\"" Sep 06 06:22:58.609629 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.609510 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dc97f9\"/\"openshift-service-ca.crt\"" Sep 06 06:22:58.614334 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.614310 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk"] Sep 06 06:22:58.752246 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.752212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-home\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.752390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.752282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4215882-fd7f-400f-91a3-832fbf21ba40-tls-certs\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.752390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.752311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz57z\" (UniqueName: \"kubernetes.io/projected/f4215882-fd7f-400f-91a3-832fbf21ba40-kube-api-access-fz57z\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.752390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.752336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-model-cache\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.752390 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.752379 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-dshm\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.752551 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.752401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-tmp-dir\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.853675 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.853589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-home\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.853824 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.853675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4215882-fd7f-400f-91a3-832fbf21ba40-tls-certs\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.853824 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.853717 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz57z\" (UniqueName: \"kubernetes.io/projected/f4215882-fd7f-400f-91a3-832fbf21ba40-kube-api-access-fz57z\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.853824 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.853753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-model-cache\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.853824 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.853781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-dshm\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.853824 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.853805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-tmp-dir\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.854132 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.854072 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-home\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.854132 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.854088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-model-cache\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.854132 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.854130 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-tmp-dir\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.855975 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.855950 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-dshm\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.856172 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.856157 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4215882-fd7f-400f-91a3-832fbf21ba40-tls-certs\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.861454 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.861426 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz57z\" (UniqueName: \"kubernetes.io/projected/f4215882-fd7f-400f-91a3-832fbf21ba40-kube-api-access-fz57z\") pod \"rate-test-model-dc97f9-kserve-cb7f75667-j8wkk\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:58.915912 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:58.915880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:22:59.187065 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:22:59.187033 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk"] Sep 06 06:22:59.188622 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:22:59.188593 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4215882_fd7f_400f_91a3_832fbf21ba40.slice/crio-8218d6c77ef2db265b9e3c004390b0385e84adffbeb56a9e3112168cd7619077 WatchSource:0}: Error finding container 8218d6c77ef2db265b9e3c004390b0385e84adffbeb56a9e3112168cd7619077: Status 404 returned error can't find the container with id 8218d6c77ef2db265b9e3c004390b0385e84adffbeb56a9e3112168cd7619077 Sep 06 06:23:00.045701 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:00.045664 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" event={"ID":"f4215882-fd7f-400f-91a3-832fbf21ba40","Type":"ContainerStarted","Data":"28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7"} Sep 06 06:23:00.046107 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:00.045714 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" event={"ID":"f4215882-fd7f-400f-91a3-832fbf21ba40","Type":"ContainerStarted","Data":"8218d6c77ef2db265b9e3c004390b0385e84adffbeb56a9e3112168cd7619077"} Sep 06 06:23:00.063863 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:00.063805 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" podStartSLOduration=2.063790673 podStartE2EDuration="2.063790673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:23:00.061867523 +0000 UTC m=+1745.710486707" watchObservedRunningTime="2026-09-06 06:23:00.063790673 +0000 UTC m=+1745.712409852" Sep 06 06:23:06.749942 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:06.749907 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv"] Sep 06 06:23:06.750419 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:06.750182 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" podUID="e2229fad-7b16-4b47-96ce-261750fc7470" containerName="payload-processing" containerID="cri-o://df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244" gracePeriod=30 Sep 06 06:23:06.760131 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:06.759817 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp"] Sep 06 06:23:06.760131 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:06.760069 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" podUID="27d5b4e3-22af-45c9-9ec0-2df27edcb4b3" containerName="payload-pre-processing" containerID="cri-o://2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158" gracePeriod=30 Sep 06 06:23:07.007813 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.007759 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" Sep 06 06:23:07.034889 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.034862 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" Sep 06 06:23:07.083902 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.083869 2580 generic.go:356] "Generic (PLEG): container finished" podID="27d5b4e3-22af-45c9-9ec0-2df27edcb4b3" containerID="2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158" exitCode=0 Sep 06 06:23:07.084061 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.083919 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" Sep 06 06:23:07.084061 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.083958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" event={"ID":"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3","Type":"ContainerDied","Data":"2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158"} Sep 06 06:23:07.084061 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.084015 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp" event={"ID":"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3","Type":"ContainerDied","Data":"5364d863a1dfa9ffaea6968236b47eb9cf2a1318ee30579a5ce050d4e4118da6"} Sep 06 06:23:07.084061 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.084045 2580 scope.go:117] "RemoveContainer" containerID="2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158" Sep 06 06:23:07.085475 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.085450 2580 generic.go:356] "Generic (PLEG): container finished" podID="e2229fad-7b16-4b47-96ce-261750fc7470" containerID="df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244" exitCode=0 Sep 06 06:23:07.085591 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.085486 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" Sep 06 06:23:07.085591 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.085530 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" event={"ID":"e2229fad-7b16-4b47-96ce-261750fc7470","Type":"ContainerDied","Data":"df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244"} Sep 06 06:23:07.085591 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.085568 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv" event={"ID":"e2229fad-7b16-4b47-96ce-261750fc7470","Type":"ContainerDied","Data":"816745d74e6f7539e0dc3c8b19ec35d71cb8dc6814b33cd150741c4332ca7a18"} Sep 06 06:23:07.093471 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.093453 2580 scope.go:117] "RemoveContainer" containerID="2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158" Sep 06 06:23:07.093723 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:23:07.093705 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158\": container with ID starting with 2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158 not found: ID does not exist" containerID="2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158" Sep 06 06:23:07.093779 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.093732 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158"} err="failed to get container status \"2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158\": rpc error: code = NotFound desc = could not find container \"2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158\": container with ID starting with 2cbf3629682d0ab9d9c2d20f2049b8bb6d03f1fafecf1df3e75398fc26524158 not found: ID does not exist" Sep 06 06:23:07.093779 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.093749 2580 scope.go:117] "RemoveContainer" containerID="df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244" Sep 06 06:23:07.101481 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.101464 2580 scope.go:117] "RemoveContainer" containerID="df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244" Sep 06 06:23:07.101723 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:23:07.101703 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244\": container with ID starting with df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244 not found: ID does not exist" containerID="df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244" Sep 06 06:23:07.101788 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.101732 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244"} err="failed to get container status \"df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244\": rpc error: code = NotFound desc = could not find container \"df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244\": container with ID starting with df57ff3fb251b693d9d4688216e55857deefbfabfadcfcbfd9d7eac8d464c244 not found: ID does not exist" Sep 06 06:23:07.133008 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.132988 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gxqz\" (UniqueName: \"kubernetes.io/projected/e2229fad-7b16-4b47-96ce-261750fc7470-kube-api-access-2gxqz\") pod \"e2229fad-7b16-4b47-96ce-261750fc7470\" (UID: \"e2229fad-7b16-4b47-96ce-261750fc7470\") " Sep 06 06:23:07.133086 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.133037 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3-plugins-config-volume\") pod \"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3\" (UID: \"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3\") " Sep 06 06:23:07.133157 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.133143 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg89j\" (UniqueName: \"kubernetes.io/projected/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3-kube-api-access-hg89j\") pod \"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3\" (UID: \"27d5b4e3-22af-45c9-9ec0-2df27edcb4b3\") " Sep 06 06:23:07.133206 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.133193 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2229fad-7b16-4b47-96ce-261750fc7470-plugins-config-volume\") pod \"e2229fad-7b16-4b47-96ce-261750fc7470\" (UID: \"e2229fad-7b16-4b47-96ce-261750fc7470\") " Sep 06 06:23:07.133473 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.133449 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27d5b4e3-22af-45c9-9ec0-2df27edcb4b3" (UID: "27d5b4e3-22af-45c9-9ec0-2df27edcb4b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:23:07.133567 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.133490 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2229fad-7b16-4b47-96ce-261750fc7470-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2229fad-7b16-4b47-96ce-261750fc7470" (UID: "e2229fad-7b16-4b47-96ce-261750fc7470"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:23:07.133567 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.133508 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:23:07.135006 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.134987 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2229fad-7b16-4b47-96ce-261750fc7470-kube-api-access-2gxqz" (OuterVolumeSpecName: "kube-api-access-2gxqz") pod "e2229fad-7b16-4b47-96ce-261750fc7470" (UID: "e2229fad-7b16-4b47-96ce-261750fc7470"). InnerVolumeSpecName "kube-api-access-2gxqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:23:07.135094 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.135080 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3-kube-api-access-hg89j" (OuterVolumeSpecName: "kube-api-access-hg89j") pod "27d5b4e3-22af-45c9-9ec0-2df27edcb4b3" (UID: "27d5b4e3-22af-45c9-9ec0-2df27edcb4b3"). InnerVolumeSpecName "kube-api-access-hg89j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:23:07.234355 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.234329 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gxqz\" (UniqueName: \"kubernetes.io/projected/e2229fad-7b16-4b47-96ce-261750fc7470-kube-api-access-2gxqz\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:23:07.234355 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.234350 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg89j\" (UniqueName: \"kubernetes.io/projected/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3-kube-api-access-hg89j\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:23:07.234355 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.234360 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2229fad-7b16-4b47-96ce-261750fc7470-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:23:07.408904 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.408876 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv"] Sep 06 06:23:07.414861 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.414823 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b7c728-84d64d6b9-knkcv"] Sep 06 06:23:07.425693 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.425672 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp"] Sep 06 06:23:07.429541 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:07.429520 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b7c728-6b96bd4bc4-m9hzp"] Sep 06 06:23:08.875448 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:08.875412 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d5b4e3-22af-45c9-9ec0-2df27edcb4b3" path="/var/lib/kubelet/pods/27d5b4e3-22af-45c9-9ec0-2df27edcb4b3/volumes" Sep 06 06:23:08.875786 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:08.875762 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2229fad-7b16-4b47-96ce-261750fc7470" path="/var/lib/kubelet/pods/e2229fad-7b16-4b47-96ce-261750fc7470/volumes" Sep 06 06:23:08.916441 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:08.916401 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:23:08.916559 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:08.916457 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:23:08.929671 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:08.929650 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:23:09.114897 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:09.114868 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:23:47.425296 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:47.425207 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk"] Sep 06 06:23:53.365841 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.365798 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh"] Sep 06 06:23:53.366281 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.366265 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d5b4e3-22af-45c9-9ec0-2df27edcb4b3" containerName="payload-pre-processing" Sep 06 06:23:53.366336 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.366283 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d5b4e3-22af-45c9-9ec0-2df27edcb4b3" containerName="payload-pre-processing" Sep 06 06:23:53.366336 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.366292 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2229fad-7b16-4b47-96ce-261750fc7470" containerName="payload-processing" Sep 06 06:23:53.366336 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.366297 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2229fad-7b16-4b47-96ce-261750fc7470" containerName="payload-processing" Sep 06 06:23:53.366431 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.366353 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2229fad-7b16-4b47-96ce-261750fc7470" containerName="payload-processing" Sep 06 06:23:53.366431 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.366364 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d5b4e3-22af-45c9-9ec0-2df27edcb4b3" containerName="payload-pre-processing" Sep 06 06:23:53.369617 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.369563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" Sep 06 06:23:53.372219 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.372195 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a36847\"" Sep 06 06:23:53.382156 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.382131 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh"] Sep 06 06:23:53.469241 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.469211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jmtj\" (UniqueName: \"kubernetes.io/projected/70a566d5-c36b-4a30-ac38-077588601917-kube-api-access-7jmtj\") pod \"payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh\" (UID: \"70a566d5-c36b-4a30-ac38-077588601917\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" Sep 06 06:23:53.469400 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.469273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a566d5-c36b-4a30-ac38-077588601917-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh\" (UID: \"70a566d5-c36b-4a30-ac38-077588601917\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" Sep 06 06:23:53.570666 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.570634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jmtj\" (UniqueName: \"kubernetes.io/projected/70a566d5-c36b-4a30-ac38-077588601917-kube-api-access-7jmtj\") pod \"payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh\" (UID: \"70a566d5-c36b-4a30-ac38-077588601917\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" Sep 06 06:23:53.570857 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.570705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a566d5-c36b-4a30-ac38-077588601917-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh\" (UID: \"70a566d5-c36b-4a30-ac38-077588601917\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" Sep 06 06:23:53.571408 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.571388 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a566d5-c36b-4a30-ac38-077588601917-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh\" (UID: \"70a566d5-c36b-4a30-ac38-077588601917\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" Sep 06 06:23:53.578946 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.578924 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jmtj\" (UniqueName: \"kubernetes.io/projected/70a566d5-c36b-4a30-ac38-077588601917-kube-api-access-7jmtj\") pod \"payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh\" (UID: \"70a566d5-c36b-4a30-ac38-077588601917\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" Sep 06 06:23:53.680938 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.680909 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" Sep 06 06:23:53.948357 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:53.948333 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh"] Sep 06 06:23:53.950674 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:23:53.950645 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a566d5_c36b_4a30_ac38_077588601917.slice/crio-65cbef31502ae4fbe18e2487308112ff9440894de9f4cda18cbdf58a4b868f94 WatchSource:0}: Error finding container 65cbef31502ae4fbe18e2487308112ff9440894de9f4cda18cbdf58a4b868f94: Status 404 returned error can't find the container with id 65cbef31502ae4fbe18e2487308112ff9440894de9f4cda18cbdf58a4b868f94 Sep 06 06:23:54.334696 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:54.334600 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" event={"ID":"70a566d5-c36b-4a30-ac38-077588601917","Type":"ContainerStarted","Data":"3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061"} Sep 06 06:23:54.334696 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:54.334661 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" event={"ID":"70a566d5-c36b-4a30-ac38-077588601917","Type":"ContainerStarted","Data":"65cbef31502ae4fbe18e2487308112ff9440894de9f4cda18cbdf58a4b868f94"} Sep 06 06:23:54.334696 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:54.334690 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" Sep 06 06:23:54.350672 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:23:54.350623 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" podStartSLOduration=1.35060798 podStartE2EDuration="1.35060798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:23:54.348672239 +0000 UTC m=+1799.997291419" watchObservedRunningTime="2026-09-06 06:23:54.35060798 +0000 UTC m=+1799.999227158" Sep 06 06:24:02.445451 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.445383 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" podUID="f4215882-fd7f-400f-91a3-832fbf21ba40" containerName="main" containerID="cri-o://28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7" gracePeriod=45 Sep 06 06:24:02.704488 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.704427 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:24:02.754865 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.754820 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-tmp-dir\") pod \"f4215882-fd7f-400f-91a3-832fbf21ba40\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " Sep 06 06:24:02.754865 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.754874 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-model-cache\") pod \"f4215882-fd7f-400f-91a3-832fbf21ba40\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " Sep 06 06:24:02.755118 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.754894 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4215882-fd7f-400f-91a3-832fbf21ba40-tls-certs\") pod \"f4215882-fd7f-400f-91a3-832fbf21ba40\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " Sep 06 06:24:02.755118 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.754931 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-dshm\") pod \"f4215882-fd7f-400f-91a3-832fbf21ba40\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " Sep 06 06:24:02.755118 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.754952 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz57z\" (UniqueName: \"kubernetes.io/projected/f4215882-fd7f-400f-91a3-832fbf21ba40-kube-api-access-fz57z\") pod \"f4215882-fd7f-400f-91a3-832fbf21ba40\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " Sep 06 06:24:02.755118 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.755004 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-home\") pod \"f4215882-fd7f-400f-91a3-832fbf21ba40\" (UID: \"f4215882-fd7f-400f-91a3-832fbf21ba40\") " Sep 06 06:24:02.755333 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.755143 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4215882-fd7f-400f-91a3-832fbf21ba40" (UID: "f4215882-fd7f-400f-91a3-832fbf21ba40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:24:02.755333 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.755151 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4215882-fd7f-400f-91a3-832fbf21ba40" (UID: "f4215882-fd7f-400f-91a3-832fbf21ba40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:24:02.755419 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.755359 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-home" (OuterVolumeSpecName: "home") pod "f4215882-fd7f-400f-91a3-832fbf21ba40" (UID: "f4215882-fd7f-400f-91a3-832fbf21ba40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:24:02.755419 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.755408 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-tmp-dir\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:24:02.755492 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.755422 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-model-cache\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:24:02.757178 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.757149 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4215882-fd7f-400f-91a3-832fbf21ba40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4215882-fd7f-400f-91a3-832fbf21ba40" (UID: "f4215882-fd7f-400f-91a3-832fbf21ba40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:24:02.757275 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.757206 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4215882-fd7f-400f-91a3-832fbf21ba40-kube-api-access-fz57z" (OuterVolumeSpecName: "kube-api-access-fz57z") pod "f4215882-fd7f-400f-91a3-832fbf21ba40" (UID: "f4215882-fd7f-400f-91a3-832fbf21ba40"). InnerVolumeSpecName "kube-api-access-fz57z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:24:02.757275 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.757218 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-dshm" (OuterVolumeSpecName: "dshm") pod "f4215882-fd7f-400f-91a3-832fbf21ba40" (UID: "f4215882-fd7f-400f-91a3-832fbf21ba40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:24:02.855997 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.855967 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4215882-fd7f-400f-91a3-832fbf21ba40-tls-certs\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:24:02.855997 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.855990 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-dshm\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:24:02.855997 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.856000 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz57z\" (UniqueName: \"kubernetes.io/projected/f4215882-fd7f-400f-91a3-832fbf21ba40-kube-api-access-fz57z\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:24:02.856188 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:02.856008 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4215882-fd7f-400f-91a3-832fbf21ba40-home\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:24:03.380843 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:03.380806 2580 generic.go:356] "Generic (PLEG): container finished" podID="f4215882-fd7f-400f-91a3-832fbf21ba40" containerID="28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7" exitCode=0 Sep 06 06:24:03.381014 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:03.380879 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" Sep 06 06:24:03.381014 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:03.380898 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" event={"ID":"f4215882-fd7f-400f-91a3-832fbf21ba40","Type":"ContainerDied","Data":"28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7"} Sep 06 06:24:03.381014 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:03.380948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk" event={"ID":"f4215882-fd7f-400f-91a3-832fbf21ba40","Type":"ContainerDied","Data":"8218d6c77ef2db265b9e3c004390b0385e84adffbeb56a9e3112168cd7619077"} Sep 06 06:24:03.381014 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:03.380971 2580 scope.go:117] "RemoveContainer" containerID="28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7" Sep 06 06:24:03.389846 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:03.389817 2580 scope.go:117] "RemoveContainer" containerID="28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7" Sep 06 06:24:03.390094 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:24:03.390077 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7\": container with ID starting with 28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7 not found: ID does not exist" containerID="28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7" Sep 06 06:24:03.390141 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:03.390101 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7"} err="failed to get container status \"28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7\": rpc error: code = NotFound desc = could not find container \"28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7\": container with ID starting with 28d4101b92daef9d65e5e2e0374dbe664b3259e0219876d97af192c4bc4101d7 not found: ID does not exist" Sep 06 06:24:03.397298 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:03.397277 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk"] Sep 06 06:24:03.400447 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:03.400426 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/rate-test-model-dc97f9-kserve-cb7f75667-j8wkk"] Sep 06 06:24:04.876001 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:04.875973 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4215882-fd7f-400f-91a3-832fbf21ba40" path="/var/lib/kubelet/pods/f4215882-fd7f-400f-91a3-832fbf21ba40/volumes" Sep 06 06:24:05.340506 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:05.340475 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" Sep 06 06:24:22.355264 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.355224 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9"] Sep 06 06:24:22.357966 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.355738 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4215882-fd7f-400f-91a3-832fbf21ba40" containerName="main" Sep 06 06:24:22.357966 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.355753 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4215882-fd7f-400f-91a3-832fbf21ba40" containerName="main" Sep 06 06:24:22.357966 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.355857 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4215882-fd7f-400f-91a3-832fbf21ba40" containerName="main" Sep 06 06:24:22.359252 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.359238 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" Sep 06 06:24:22.366266 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.366242 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a52b50-dockercfg-94rlr\"" Sep 06 06:24:22.366458 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.366439 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a52b50\"" Sep 06 06:24:22.373816 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.373791 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9"] Sep 06 06:24:22.544263 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.544222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-962xd\" (UniqueName: \"kubernetes.io/projected/42a18e35-4ca4-4784-ba22-873e5272e440-kube-api-access-962xd\") pod \"payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9\" (UID: \"42a18e35-4ca4-4784-ba22-873e5272e440\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" Sep 06 06:24:22.544469 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.544290 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a18e35-4ca4-4784-ba22-873e5272e440-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9\" (UID: \"42a18e35-4ca4-4784-ba22-873e5272e440\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" Sep 06 06:24:22.645419 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.645332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-962xd\" (UniqueName: \"kubernetes.io/projected/42a18e35-4ca4-4784-ba22-873e5272e440-kube-api-access-962xd\") pod \"payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9\" (UID: \"42a18e35-4ca4-4784-ba22-873e5272e440\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" Sep 06 06:24:22.645568 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.645417 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a18e35-4ca4-4784-ba22-873e5272e440-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9\" (UID: \"42a18e35-4ca4-4784-ba22-873e5272e440\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" Sep 06 06:24:22.646133 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.646115 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a18e35-4ca4-4784-ba22-873e5272e440-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9\" (UID: \"42a18e35-4ca4-4784-ba22-873e5272e440\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" Sep 06 06:24:22.654530 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.654507 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-962xd\" (UniqueName: \"kubernetes.io/projected/42a18e35-4ca4-4784-ba22-873e5272e440-kube-api-access-962xd\") pod \"payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9\" (UID: \"42a18e35-4ca4-4784-ba22-873e5272e440\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" Sep 06 06:24:22.669334 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.669312 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" Sep 06 06:24:22.943066 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:22.943042 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9"] Sep 06 06:24:22.944675 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:24:22.944652 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a18e35_4ca4_4784_ba22_873e5272e440.slice/crio-0b108db18f3196c5f4aed2de6b6b891d3b17a4e02545b57af22cfb2865574969 WatchSource:0}: Error finding container 0b108db18f3196c5f4aed2de6b6b891d3b17a4e02545b57af22cfb2865574969: Status 404 returned error can't find the container with id 0b108db18f3196c5f4aed2de6b6b891d3b17a4e02545b57af22cfb2865574969 Sep 06 06:24:23.484550 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:23.484514 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" event={"ID":"42a18e35-4ca4-4784-ba22-873e5272e440","Type":"ContainerStarted","Data":"63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32"} Sep 06 06:24:23.484934 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:23.484570 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" event={"ID":"42a18e35-4ca4-4784-ba22-873e5272e440","Type":"ContainerStarted","Data":"0b108db18f3196c5f4aed2de6b6b891d3b17a4e02545b57af22cfb2865574969"} Sep 06 06:24:23.484934 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:23.484713 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" Sep 06 06:24:34.491766 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:34.491731 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" Sep 06 06:24:34.508548 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:34.508487 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" podStartSLOduration=12.508469461 podStartE2EDuration="12.508469461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:24:23.503311394 +0000 UTC m=+1829.151930575" watchObservedRunningTime="2026-09-06 06:24:34.508469461 +0000 UTC m=+1840.157088637" Sep 06 06:24:50.597637 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:50.597602 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-95mv6_287f1091-aa00-4497-badd-63c872f52fd6/payload-processing/0.log" Sep 06 06:24:55.995642 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:55.995599 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp"] Sep 06 06:24:56.000644 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.000615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.004540 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.004514 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a36847\"/\"default-dockercfg-njzbz\"" Sep 06 06:24:56.004667 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.004562 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a36847\"/\"kube-root-ca.crt\"" Sep 06 06:24:56.004667 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.004600 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a36847\"/\"openshift-service-ca.crt\"" Sep 06 06:24:56.004667 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.004651 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a36847\"/\"ipp-route-a36847-kserve-self-signed-certs\"" Sep 06 06:24:56.010616 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.010596 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp"] Sep 06 06:24:56.156579 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.156543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baa44881-b235-4e82-869f-977eab080e46-tls-certs\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.156579 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.156581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnqd4\" (UniqueName: \"kubernetes.io/projected/baa44881-b235-4e82-869f-977eab080e46-kube-api-access-hnqd4\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.156865 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.156705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-tmp-dir\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.156865 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.156758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-model-cache\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.156865 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.156813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-home\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.157031 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.156883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-dshm\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.258094 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.258007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-dshm\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.258239 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.258101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baa44881-b235-4e82-869f-977eab080e46-tls-certs\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.258239 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.258127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnqd4\" (UniqueName: \"kubernetes.io/projected/baa44881-b235-4e82-869f-977eab080e46-kube-api-access-hnqd4\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.258239 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.258190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-tmp-dir\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.258422 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.258242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-model-cache\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.258422 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.258295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-home\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.258604 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.258583 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-tmp-dir\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.258671 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.258628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-model-cache\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.258709 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.258669 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-home\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.260364 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.260342 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-dshm\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.260544 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.260527 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baa44881-b235-4e82-869f-977eab080e46-tls-certs\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.265543 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.265523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnqd4\" (UniqueName: \"kubernetes.io/projected/baa44881-b235-4e82-869f-977eab080e46-kube-api-access-hnqd4\") pod \"ipp-route-a36847-kserve-54885979ff-6p8tp\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.310729 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.310695 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:24:56.577543 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.577517 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp"] Sep 06 06:24:56.578917 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:24:56.578881 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaa44881_b235_4e82_869f_977eab080e46.slice/crio-20d386f447d2c6a30ba6e074ce617f2b166c8bc5be8053d4f89bd3d075a4cbf6 WatchSource:0}: Error finding container 20d386f447d2c6a30ba6e074ce617f2b166c8bc5be8053d4f89bd3d075a4cbf6: Status 404 returned error can't find the container with id 20d386f447d2c6a30ba6e074ce617f2b166c8bc5be8053d4f89bd3d075a4cbf6 Sep 06 06:24:56.660951 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.660925 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" event={"ID":"baa44881-b235-4e82-869f-977eab080e46","Type":"ContainerStarted","Data":"31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2"} Sep 06 06:24:56.661067 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.660968 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" event={"ID":"baa44881-b235-4e82-869f-977eab080e46","Type":"ContainerStarted","Data":"20d386f447d2c6a30ba6e074ce617f2b166c8bc5be8053d4f89bd3d075a4cbf6"} Sep 06 06:24:56.677165 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:24:56.677123 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" podStartSLOduration=1.677106868 podStartE2EDuration="1.677106868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:24:56.676210129 +0000 UTC m=+1862.324829330" watchObservedRunningTime="2026-09-06 06:24:56.677106868 +0000 UTC m=+1862.325726045" Sep 06 06:25:06.311628 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:06.311587 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:25:06.312083 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:06.311666 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:25:06.324217 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:06.324190 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:25:06.727944 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:06.727911 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:25:23.284869 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:23.284767 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-95mv6_287f1091-aa00-4497-badd-63c872f52fd6/payload-processing/0.log" Sep 06 06:25:28.865883 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:28.865844 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c"] Sep 06 06:25:28.869536 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:28.869514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" Sep 06 06:25:28.871930 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:28.871901 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-3cf67d-dockercfg-5tqnk\"" Sep 06 06:25:28.872056 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:28.871934 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-3cf67d\"" Sep 06 06:25:28.879074 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:28.879053 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c"] Sep 06 06:25:28.901334 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:28.901305 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j"] Sep 06 06:25:28.905438 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:28.905418 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" Sep 06 06:25:28.913236 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:28.913215 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j"] Sep 06 06:25:28.957278 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:28.957246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j\" (UID: \"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" Sep 06 06:25:28.957432 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:28.957291 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd4fb06-4ed1-4a25-8c60-9acef480344a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c\" (UID: \"8dd4fb06-4ed1-4a25-8c60-9acef480344a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" Sep 06 06:25:28.957432 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:28.957366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xr9g\" (UniqueName: \"kubernetes.io/projected/8dd4fb06-4ed1-4a25-8c60-9acef480344a-kube-api-access-4xr9g\") pod \"payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c\" (UID: \"8dd4fb06-4ed1-4a25-8c60-9acef480344a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" Sep 06 06:25:28.957506 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:28.957427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvxwc\" (UniqueName: \"kubernetes.io/projected/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb-kube-api-access-pvxwc\") pod \"payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j\" (UID: \"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" Sep 06 06:25:29.058910 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.058873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j\" (UID: \"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" Sep 06 06:25:29.058910 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.058919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd4fb06-4ed1-4a25-8c60-9acef480344a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c\" (UID: \"8dd4fb06-4ed1-4a25-8c60-9acef480344a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" Sep 06 06:25:29.059165 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.058957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xr9g\" (UniqueName: \"kubernetes.io/projected/8dd4fb06-4ed1-4a25-8c60-9acef480344a-kube-api-access-4xr9g\") pod \"payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c\" (UID: \"8dd4fb06-4ed1-4a25-8c60-9acef480344a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" Sep 06 06:25:29.059165 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.058982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvxwc\" (UniqueName: \"kubernetes.io/projected/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb-kube-api-access-pvxwc\") pod \"payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j\" (UID: \"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" Sep 06 06:25:29.059662 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.059629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j\" (UID: \"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" Sep 06 06:25:29.059731 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.059702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd4fb06-4ed1-4a25-8c60-9acef480344a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c\" (UID: \"8dd4fb06-4ed1-4a25-8c60-9acef480344a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" Sep 06 06:25:29.066899 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.066872 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvxwc\" (UniqueName: \"kubernetes.io/projected/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb-kube-api-access-pvxwc\") pod \"payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j\" (UID: \"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" Sep 06 06:25:29.067006 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.066929 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xr9g\" (UniqueName: \"kubernetes.io/projected/8dd4fb06-4ed1-4a25-8c60-9acef480344a-kube-api-access-4xr9g\") pod \"payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c\" (UID: \"8dd4fb06-4ed1-4a25-8c60-9acef480344a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" Sep 06 06:25:29.181497 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.181464 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" Sep 06 06:25:29.218296 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.218260 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" Sep 06 06:25:29.453112 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.453087 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c"] Sep 06 06:25:29.454762 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:25:29.454735 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd4fb06_4ed1_4a25_8c60_9acef480344a.slice/crio-b4264bfa78262db929bfa758f3054f7921900d5b0973a9acb1cb72c83ac8f852 WatchSource:0}: Error finding container b4264bfa78262db929bfa758f3054f7921900d5b0973a9acb1cb72c83ac8f852: Status 404 returned error can't find the container with id b4264bfa78262db929bfa758f3054f7921900d5b0973a9acb1cb72c83ac8f852 Sep 06 06:25:29.495866 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.495821 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j"] Sep 06 06:25:29.498630 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:25:29.498600 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7d9164_0d4b_4f2c_b70d_4d8abb585deb.slice/crio-08190dbe6191e372c0d08cfc51da39d88cf71061b3866075caaa8f0b0e382920 WatchSource:0}: Error finding container 08190dbe6191e372c0d08cfc51da39d88cf71061b3866075caaa8f0b0e382920: Status 404 returned error can't find the container with id 08190dbe6191e372c0d08cfc51da39d88cf71061b3866075caaa8f0b0e382920 Sep 06 06:25:29.835145 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.835054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" event={"ID":"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb","Type":"ContainerStarted","Data":"a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943"} Sep 06 06:25:29.835311 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.835210 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" Sep 06 06:25:29.835311 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.835253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" event={"ID":"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb","Type":"ContainerStarted","Data":"08190dbe6191e372c0d08cfc51da39d88cf71061b3866075caaa8f0b0e382920"} Sep 06 06:25:29.836712 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.836676 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" event={"ID":"8dd4fb06-4ed1-4a25-8c60-9acef480344a","Type":"ContainerStarted","Data":"e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76"} Sep 06 06:25:29.836887 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.836724 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" event={"ID":"8dd4fb06-4ed1-4a25-8c60-9acef480344a","Type":"ContainerStarted","Data":"b4264bfa78262db929bfa758f3054f7921900d5b0973a9acb1cb72c83ac8f852"} Sep 06 06:25:29.836887 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.836752 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" Sep 06 06:25:29.853733 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.853670 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" podStartSLOduration=1.8536554010000001 podStartE2EDuration="1.853655401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:25:29.851305892 +0000 UTC m=+1895.499925073" watchObservedRunningTime="2026-09-06 06:25:29.853655401 +0000 UTC m=+1895.502274581" Sep 06 06:25:29.866113 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:29.866063 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" podStartSLOduration=1.866046581 podStartE2EDuration="1.866046581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:25:29.865219223 +0000 UTC m=+1895.513838401" watchObservedRunningTime="2026-09-06 06:25:29.866046581 +0000 UTC m=+1895.514665761" Sep 06 06:25:40.843759 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:40.843727 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" Sep 06 06:25:40.844135 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:40.843871 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" Sep 06 06:25:52.616814 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:52.616778 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72"] Sep 06 06:25:52.621018 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:52.620997 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72" Sep 06 06:25:52.623965 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:52.623927 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:25:52.625022 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:52.625000 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:25:52.626856 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:52.626813 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72"] Sep 06 06:25:52.686001 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:52.685969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d61d61-b55c-47df-aba9-302d220ee53b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72\" (UID: \"c7d61d61-b55c-47df-aba9-302d220ee53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72" Sep 06 06:25:52.786818 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:52.786782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d61d61-b55c-47df-aba9-302d220ee53b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72\" (UID: \"c7d61d61-b55c-47df-aba9-302d220ee53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72" Sep 06 06:25:52.787417 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:52.787396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d61d61-b55c-47df-aba9-302d220ee53b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72\" (UID: \"c7d61d61-b55c-47df-aba9-302d220ee53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72" Sep 06 06:25:52.934559 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:52.934527 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72" Sep 06 06:25:53.203113 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:53.203084 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72"] Sep 06 06:25:53.204767 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:25:53.204740 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d61d61_b55c_47df_aba9_302d220ee53b.slice/crio-053aec166aa1344566114fffdf7be870e4b7ed54a0f31d85425fbafa53d65bcd WatchSource:0}: Error finding container 053aec166aa1344566114fffdf7be870e4b7ed54a0f31d85425fbafa53d65bcd: Status 404 returned error can't find the container with id 053aec166aa1344566114fffdf7be870e4b7ed54a0f31d85425fbafa53d65bcd Sep 06 06:25:53.971345 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:53.971310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72" event={"ID":"c7d61d61-b55c-47df-aba9-302d220ee53b","Type":"ContainerStarted","Data":"593946e85d57ab4daa1e4274165bb06f99e03ff4148664fb1451a6e14acdb19c"} Sep 06 06:25:53.971881 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:53.971368 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72" event={"ID":"c7d61d61-b55c-47df-aba9-302d220ee53b","Type":"ContainerStarted","Data":"053aec166aa1344566114fffdf7be870e4b7ed54a0f31d85425fbafa53d65bcd"} Sep 06 06:25:53.987433 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:53.987385 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72" podStartSLOduration=1.9873705529999999 podStartE2EDuration="1.987370553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:25:53.984677263 +0000 UTC m=+1919.633296442" watchObservedRunningTime="2026-09-06 06:25:53.987370553 +0000 UTC m=+1919.635989732" Sep 06 06:25:54.985097 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:54.985063 2580 generic.go:356] "Generic (PLEG): container finished" podID="c7d61d61-b55c-47df-aba9-302d220ee53b" containerID="593946e85d57ab4daa1e4274165bb06f99e03ff4148664fb1451a6e14acdb19c" exitCode=0 Sep 06 06:25:54.985530 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:54.985150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72" event={"ID":"c7d61d61-b55c-47df-aba9-302d220ee53b","Type":"ContainerDied","Data":"593946e85d57ab4daa1e4274165bb06f99e03ff4148664fb1451a6e14acdb19c"} Sep 06 06:25:56.125615 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:56.125592 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72" Sep 06 06:25:56.219731 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:56.219697 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d61d61-b55c-47df-aba9-302d220ee53b-maas-api-service-ca\") pod \"c7d61d61-b55c-47df-aba9-302d220ee53b\" (UID: \"c7d61d61-b55c-47df-aba9-302d220ee53b\") " Sep 06 06:25:56.220055 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:56.220029 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d61d61-b55c-47df-aba9-302d220ee53b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7d61d61-b55c-47df-aba9-302d220ee53b" (UID: "c7d61d61-b55c-47df-aba9-302d220ee53b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:25:56.320583 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:56.320515 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d61d61-b55c-47df-aba9-302d220ee53b-maas-api-service-ca\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:25:56.998980 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:56.998950 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72" Sep 06 06:25:56.999156 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:56.998991 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72" event={"ID":"c7d61d61-b55c-47df-aba9-302d220ee53b","Type":"ContainerDied","Data":"053aec166aa1344566114fffdf7be870e4b7ed54a0f31d85425fbafa53d65bcd"} Sep 06 06:25:56.999156 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:56.999044 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="053aec166aa1344566114fffdf7be870e4b7ed54a0f31d85425fbafa53d65bcd" Sep 06 06:25:58.683453 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:58.683419 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c"] Sep 06 06:25:58.683933 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:58.683627 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" podUID="8dd4fb06-4ed1-4a25-8c60-9acef480344a" containerName="payload-processing" containerID="cri-o://e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76" gracePeriod=30 Sep 06 06:25:58.694571 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:58.694542 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j"] Sep 06 06:25:58.694812 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:58.694785 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" podUID="1a7d9164-0d4b-4f2c-b70d-4d8abb585deb" containerName="payload-pre-processing" containerID="cri-o://a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943" gracePeriod=30 Sep 06 06:25:58.844777 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:58.844742 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72"] Sep 06 06:25:58.847549 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:58.847469 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3cf67d-2dc72"] Sep 06 06:25:58.877512 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:58.877481 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d61d61-b55c-47df-aba9-302d220ee53b" path="/var/lib/kubelet/pods/c7d61d61-b55c-47df-aba9-302d220ee53b/volumes" Sep 06 06:25:58.966573 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:58.966550 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" Sep 06 06:25:58.969139 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:58.969122 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" Sep 06 06:25:59.011558 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.011526 2580 generic.go:356] "Generic (PLEG): container finished" podID="1a7d9164-0d4b-4f2c-b70d-4d8abb585deb" containerID="a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943" exitCode=0 Sep 06 06:25:59.011699 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.011564 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" Sep 06 06:25:59.011699 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.011563 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" event={"ID":"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb","Type":"ContainerDied","Data":"a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943"} Sep 06 06:25:59.011699 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.011686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j" event={"ID":"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb","Type":"ContainerDied","Data":"08190dbe6191e372c0d08cfc51da39d88cf71061b3866075caaa8f0b0e382920"} Sep 06 06:25:59.011902 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.011713 2580 scope.go:117] "RemoveContainer" containerID="a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943" Sep 06 06:25:59.013086 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.013063 2580 generic.go:356] "Generic (PLEG): container finished" podID="8dd4fb06-4ed1-4a25-8c60-9acef480344a" containerID="e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76" exitCode=0 Sep 06 06:25:59.013190 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.013091 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" event={"ID":"8dd4fb06-4ed1-4a25-8c60-9acef480344a","Type":"ContainerDied","Data":"e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76"} Sep 06 06:25:59.013190 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.013105 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" Sep 06 06:25:59.013190 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.013142 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c" event={"ID":"8dd4fb06-4ed1-4a25-8c60-9acef480344a","Type":"ContainerDied","Data":"b4264bfa78262db929bfa758f3054f7921900d5b0973a9acb1cb72c83ac8f852"} Sep 06 06:25:59.021079 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.021062 2580 scope.go:117] "RemoveContainer" containerID="a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943" Sep 06 06:25:59.021342 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:25:59.021324 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943\": container with ID starting with a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943 not found: ID does not exist" containerID="a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943" Sep 06 06:25:59.021387 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.021350 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943"} err="failed to get container status \"a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943\": rpc error: code = NotFound desc = could not find container \"a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943\": container with ID starting with a3a3e253764d57e57e8475c20ddfea3223c5aa790f16a4bf50e11e118bb4b943 not found: ID does not exist" Sep 06 06:25:59.021387 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.021366 2580 scope.go:117] "RemoveContainer" containerID="e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76" Sep 06 06:25:59.029106 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.029090 2580 scope.go:117] "RemoveContainer" containerID="e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76" Sep 06 06:25:59.029337 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:25:59.029320 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76\": container with ID starting with e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76 not found: ID does not exist" containerID="e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76" Sep 06 06:25:59.029386 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.029343 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76"} err="failed to get container status \"e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76\": rpc error: code = NotFound desc = could not find container \"e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76\": container with ID starting with e2e70db4ccb12ae3a4de847139e6546f42b1763544418c9002eb1a845c3cba76 not found: ID does not exist" Sep 06 06:25:59.047677 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.047653 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xr9g\" (UniqueName: \"kubernetes.io/projected/8dd4fb06-4ed1-4a25-8c60-9acef480344a-kube-api-access-4xr9g\") pod \"8dd4fb06-4ed1-4a25-8c60-9acef480344a\" (UID: \"8dd4fb06-4ed1-4a25-8c60-9acef480344a\") " Sep 06 06:25:59.047777 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.047704 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd4fb06-4ed1-4a25-8c60-9acef480344a-plugins-config-volume\") pod \"8dd4fb06-4ed1-4a25-8c60-9acef480344a\" (UID: \"8dd4fb06-4ed1-4a25-8c60-9acef480344a\") " Sep 06 06:25:59.047777 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.047732 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb-plugins-config-volume\") pod \"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb\" (UID: \"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb\") " Sep 06 06:25:59.047908 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.047791 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvxwc\" (UniqueName: \"kubernetes.io/projected/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb-kube-api-access-pvxwc\") pod \"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb\" (UID: \"1a7d9164-0d4b-4f2c-b70d-4d8abb585deb\") " Sep 06 06:25:59.048134 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.048113 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a7d9164-0d4b-4f2c-b70d-4d8abb585deb" (UID: "1a7d9164-0d4b-4f2c-b70d-4d8abb585deb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:25:59.048197 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.048150 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd4fb06-4ed1-4a25-8c60-9acef480344a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8dd4fb06-4ed1-4a25-8c60-9acef480344a" (UID: "8dd4fb06-4ed1-4a25-8c60-9acef480344a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:25:59.049601 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.049582 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd4fb06-4ed1-4a25-8c60-9acef480344a-kube-api-access-4xr9g" (OuterVolumeSpecName: "kube-api-access-4xr9g") pod "8dd4fb06-4ed1-4a25-8c60-9acef480344a" (UID: "8dd4fb06-4ed1-4a25-8c60-9acef480344a"). InnerVolumeSpecName "kube-api-access-4xr9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:25:59.049840 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.049811 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb-kube-api-access-pvxwc" (OuterVolumeSpecName: "kube-api-access-pvxwc") pod "1a7d9164-0d4b-4f2c-b70d-4d8abb585deb" (UID: "1a7d9164-0d4b-4f2c-b70d-4d8abb585deb"). InnerVolumeSpecName "kube-api-access-pvxwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:25:59.149050 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.149024 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd4fb06-4ed1-4a25-8c60-9acef480344a-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:25:59.149050 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.149046 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:25:59.149181 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.149057 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvxwc\" (UniqueName: \"kubernetes.io/projected/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb-kube-api-access-pvxwc\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:25:59.149181 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.149065 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xr9g\" (UniqueName: \"kubernetes.io/projected/8dd4fb06-4ed1-4a25-8c60-9acef480344a-kube-api-access-4xr9g\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:25:59.336728 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.336692 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c"] Sep 06 06:25:59.342311 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.342283 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3cf67d-5789db568f-f5j2c"] Sep 06 06:25:59.351857 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.351820 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j"] Sep 06 06:25:59.357449 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:25:59.357429 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3cf67d-dfbb4995d-wgk8j"] Sep 06 06:26:00.875859 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:00.875811 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a7d9164-0d4b-4f2c-b70d-4d8abb585deb" path="/var/lib/kubelet/pods/1a7d9164-0d4b-4f2c-b70d-4d8abb585deb/volumes" Sep 06 06:26:00.876199 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:00.876173 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dd4fb06-4ed1-4a25-8c60-9acef480344a" path="/var/lib/kubelet/pods/8dd4fb06-4ed1-4a25-8c60-9acef480344a/volumes" Sep 06 06:26:08.728780 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.728746 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l"] Sep 06 06:26:08.729195 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.729182 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7d9164-0d4b-4f2c-b70d-4d8abb585deb" containerName="payload-pre-processing" Sep 06 06:26:08.729195 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.729196 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7d9164-0d4b-4f2c-b70d-4d8abb585deb" containerName="payload-pre-processing" Sep 06 06:26:08.729270 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.729209 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d61d61-b55c-47df-aba9-302d220ee53b" containerName="revoke-keys" Sep 06 06:26:08.729270 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.729215 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d61d61-b55c-47df-aba9-302d220ee53b" containerName="revoke-keys" Sep 06 06:26:08.729270 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.729232 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd4fb06-4ed1-4a25-8c60-9acef480344a" containerName="payload-processing" Sep 06 06:26:08.729270 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.729238 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd4fb06-4ed1-4a25-8c60-9acef480344a" containerName="payload-processing" Sep 06 06:26:08.729399 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.729304 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d61d61-b55c-47df-aba9-302d220ee53b" containerName="revoke-keys" Sep 06 06:26:08.729399 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.729314 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dd4fb06-4ed1-4a25-8c60-9acef480344a" containerName="payload-processing" Sep 06 06:26:08.729399 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.729321 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7d9164-0d4b-4f2c-b70d-4d8abb585deb" containerName="payload-pre-processing" Sep 06 06:26:08.733748 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.733730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l" Sep 06 06:26:08.736573 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.736548 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:26:08.737962 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.737643 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:26:08.739206 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.739184 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l"] Sep 06 06:26:08.831018 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.830986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c28c5c-8d17-43f0-abfc-0c0a82609034-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l\" (UID: \"29c28c5c-8d17-43f0-abfc-0c0a82609034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l" Sep 06 06:26:08.932173 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.932093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c28c5c-8d17-43f0-abfc-0c0a82609034-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l\" (UID: \"29c28c5c-8d17-43f0-abfc-0c0a82609034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l" Sep 06 06:26:08.932767 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:08.932739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c28c5c-8d17-43f0-abfc-0c0a82609034-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l\" (UID: \"29c28c5c-8d17-43f0-abfc-0c0a82609034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l" Sep 06 06:26:09.046893 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:09.046787 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l" Sep 06 06:26:09.314585 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:09.314506 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l"] Sep 06 06:26:09.317070 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:26:09.317041 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c28c5c_8d17_43f0_abfc_0c0a82609034.slice/crio-4c32f9ee2b2c0bd437692b9c7476a7c16a25feea1d1a102dd0392feaec08562f WatchSource:0}: Error finding container 4c32f9ee2b2c0bd437692b9c7476a7c16a25feea1d1a102dd0392feaec08562f: Status 404 returned error can't find the container with id 4c32f9ee2b2c0bd437692b9c7476a7c16a25feea1d1a102dd0392feaec08562f Sep 06 06:26:10.073122 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:10.073082 2580 generic.go:356] "Generic (PLEG): container finished" podID="29c28c5c-8d17-43f0-abfc-0c0a82609034" containerID="ddbd4f6f80a9fae2aa6594b70524bdd916bdc5ad8c1c33ddf6ba753c9046275f" exitCode=0 Sep 06 06:26:10.073502 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:10.073137 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l" event={"ID":"29c28c5c-8d17-43f0-abfc-0c0a82609034","Type":"ContainerDied","Data":"ddbd4f6f80a9fae2aa6594b70524bdd916bdc5ad8c1c33ddf6ba753c9046275f"} Sep 06 06:26:10.073502 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:10.073206 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l" event={"ID":"29c28c5c-8d17-43f0-abfc-0c0a82609034","Type":"ContainerStarted","Data":"4c32f9ee2b2c0bd437692b9c7476a7c16a25feea1d1a102dd0392feaec08562f"} Sep 06 06:26:11.211474 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:11.211448 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l" Sep 06 06:26:11.254959 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:11.254934 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c28c5c-8d17-43f0-abfc-0c0a82609034-maas-api-service-ca\") pod \"29c28c5c-8d17-43f0-abfc-0c0a82609034\" (UID: \"29c28c5c-8d17-43f0-abfc-0c0a82609034\") " Sep 06 06:26:11.255313 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:11.255293 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c28c5c-8d17-43f0-abfc-0c0a82609034-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "29c28c5c-8d17-43f0-abfc-0c0a82609034" (UID: "29c28c5c-8d17-43f0-abfc-0c0a82609034"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:26:11.356339 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:11.356271 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c28c5c-8d17-43f0-abfc-0c0a82609034-maas-api-service-ca\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:26:12.085794 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:12.085758 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l" event={"ID":"29c28c5c-8d17-43f0-abfc-0c0a82609034","Type":"ContainerDied","Data":"4c32f9ee2b2c0bd437692b9c7476a7c16a25feea1d1a102dd0392feaec08562f"} Sep 06 06:26:12.085964 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:12.085809 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c32f9ee2b2c0bd437692b9c7476a7c16a25feea1d1a102dd0392feaec08562f" Sep 06 06:26:12.085964 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:12.085770 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l" Sep 06 06:26:24.276367 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:24.276330 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh"] Sep 06 06:26:24.276813 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:24.276578 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" podUID="70a566d5-c36b-4a30-ac38-077588601917" containerName="payload-pre-processing" containerID="cri-o://3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061" gracePeriod=30 Sep 06 06:26:24.431510 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:24.431476 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l"] Sep 06 06:26:24.434514 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:24.434486 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a36847-mvz7l"] Sep 06 06:26:24.531162 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:24.531111 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" Sep 06 06:26:24.680499 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:24.680472 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a566d5-c36b-4a30-ac38-077588601917-plugins-config-volume\") pod \"70a566d5-c36b-4a30-ac38-077588601917\" (UID: \"70a566d5-c36b-4a30-ac38-077588601917\") " Sep 06 06:26:24.680635 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:24.680547 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jmtj\" (UniqueName: \"kubernetes.io/projected/70a566d5-c36b-4a30-ac38-077588601917-kube-api-access-7jmtj\") pod \"70a566d5-c36b-4a30-ac38-077588601917\" (UID: \"70a566d5-c36b-4a30-ac38-077588601917\") " Sep 06 06:26:24.680814 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:24.680791 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a566d5-c36b-4a30-ac38-077588601917-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70a566d5-c36b-4a30-ac38-077588601917" (UID: "70a566d5-c36b-4a30-ac38-077588601917"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:26:24.682598 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:24.682577 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a566d5-c36b-4a30-ac38-077588601917-kube-api-access-7jmtj" (OuterVolumeSpecName: "kube-api-access-7jmtj") pod "70a566d5-c36b-4a30-ac38-077588601917" (UID: "70a566d5-c36b-4a30-ac38-077588601917"). InnerVolumeSpecName "kube-api-access-7jmtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:26:24.781613 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:24.781548 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jmtj\" (UniqueName: \"kubernetes.io/projected/70a566d5-c36b-4a30-ac38-077588601917-kube-api-access-7jmtj\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:26:24.781613 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:24.781573 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a566d5-c36b-4a30-ac38-077588601917-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:26:24.875628 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:24.875590 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c28c5c-8d17-43f0-abfc-0c0a82609034" path="/var/lib/kubelet/pods/29c28c5c-8d17-43f0-abfc-0c0a82609034/volumes" Sep 06 06:26:25.155308 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:25.155215 2580 generic.go:356] "Generic (PLEG): container finished" podID="70a566d5-c36b-4a30-ac38-077588601917" containerID="3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061" exitCode=0 Sep 06 06:26:25.155308 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:25.155266 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" Sep 06 06:26:25.155308 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:25.155293 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" event={"ID":"70a566d5-c36b-4a30-ac38-077588601917","Type":"ContainerDied","Data":"3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061"} Sep 06 06:26:25.155599 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:25.155347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh" event={"ID":"70a566d5-c36b-4a30-ac38-077588601917","Type":"ContainerDied","Data":"65cbef31502ae4fbe18e2487308112ff9440894de9f4cda18cbdf58a4b868f94"} Sep 06 06:26:25.155599 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:25.155375 2580 scope.go:117] "RemoveContainer" containerID="3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061" Sep 06 06:26:25.164596 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:25.164575 2580 scope.go:117] "RemoveContainer" containerID="3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061" Sep 06 06:26:25.164893 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:26:25.164873 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061\": container with ID starting with 3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061 not found: ID does not exist" containerID="3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061" Sep 06 06:26:25.164966 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:25.164902 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061"} err="failed to get container status \"3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061\": rpc error: code = NotFound desc = could not find container \"3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061\": container with ID starting with 3f221854e7dfa718c722a779014b5dd72e4333fd2ae5c9322ef738d9ed674061 not found: ID does not exist" Sep 06 06:26:25.172502 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:25.172481 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh"] Sep 06 06:26:25.175611 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:25.175591 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a36847-6f6f4694f8-547kh"] Sep 06 06:26:26.875238 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:26.875206 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a566d5-c36b-4a30-ac38-077588601917" path="/var/lib/kubelet/pods/70a566d5-c36b-4a30-ac38-077588601917/volumes" Sep 06 06:26:29.769852 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:29.769805 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp"] Sep 06 06:26:44.790165 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:44.790106 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" podUID="baa44881-b235-4e82-869f-977eab080e46" containerName="main" containerID="cri-o://31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2" gracePeriod=45 Sep 06 06:26:45.043326 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.043270 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:26:45.052820 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.052798 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnqd4\" (UniqueName: \"kubernetes.io/projected/baa44881-b235-4e82-869f-977eab080e46-kube-api-access-hnqd4\") pod \"baa44881-b235-4e82-869f-977eab080e46\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " Sep 06 06:26:45.052916 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.052858 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baa44881-b235-4e82-869f-977eab080e46-tls-certs\") pod \"baa44881-b235-4e82-869f-977eab080e46\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " Sep 06 06:26:45.052916 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.052897 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-home\") pod \"baa44881-b235-4e82-869f-977eab080e46\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " Sep 06 06:26:45.053023 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.052916 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-tmp-dir\") pod \"baa44881-b235-4e82-869f-977eab080e46\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " Sep 06 06:26:45.053023 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.052945 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-model-cache\") pod \"baa44881-b235-4e82-869f-977eab080e46\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " Sep 06 06:26:45.053023 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.052983 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-dshm\") pod \"baa44881-b235-4e82-869f-977eab080e46\" (UID: \"baa44881-b235-4e82-869f-977eab080e46\") " Sep 06 06:26:45.053284 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.053202 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-home" (OuterVolumeSpecName: "home") pod "baa44881-b235-4e82-869f-977eab080e46" (UID: "baa44881-b235-4e82-869f-977eab080e46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:26:45.053284 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.053244 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-model-cache" (OuterVolumeSpecName: "model-cache") pod "baa44881-b235-4e82-869f-977eab080e46" (UID: "baa44881-b235-4e82-869f-977eab080e46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:26:45.053284 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.053218 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "baa44881-b235-4e82-869f-977eab080e46" (UID: "baa44881-b235-4e82-869f-977eab080e46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:26:45.053473 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.053317 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-home\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:26:45.053473 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.053334 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-tmp-dir\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:26:45.053473 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.053342 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-model-cache\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:26:45.055275 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.055240 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-dshm" (OuterVolumeSpecName: "dshm") pod "baa44881-b235-4e82-869f-977eab080e46" (UID: "baa44881-b235-4e82-869f-977eab080e46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:26:45.055421 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.055402 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa44881-b235-4e82-869f-977eab080e46-kube-api-access-hnqd4" (OuterVolumeSpecName: "kube-api-access-hnqd4") pod "baa44881-b235-4e82-869f-977eab080e46" (UID: "baa44881-b235-4e82-869f-977eab080e46"). InnerVolumeSpecName "kube-api-access-hnqd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:26:45.055484 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.055414 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa44881-b235-4e82-869f-977eab080e46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "baa44881-b235-4e82-869f-977eab080e46" (UID: "baa44881-b235-4e82-869f-977eab080e46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:26:45.154729 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.154684 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baa44881-b235-4e82-869f-977eab080e46-tls-certs\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:26:45.154729 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.154722 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baa44881-b235-4e82-869f-977eab080e46-dshm\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:26:45.154729 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.154732 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnqd4\" (UniqueName: \"kubernetes.io/projected/baa44881-b235-4e82-869f-977eab080e46-kube-api-access-hnqd4\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:26:45.268280 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.268241 2580 generic.go:356] "Generic (PLEG): container finished" podID="baa44881-b235-4e82-869f-977eab080e46" containerID="31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2" exitCode=0 Sep 06 06:26:45.268419 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.268290 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" Sep 06 06:26:45.268419 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.268326 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" event={"ID":"baa44881-b235-4e82-869f-977eab080e46","Type":"ContainerDied","Data":"31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2"} Sep 06 06:26:45.268419 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.268369 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp" event={"ID":"baa44881-b235-4e82-869f-977eab080e46","Type":"ContainerDied","Data":"20d386f447d2c6a30ba6e074ce617f2b166c8bc5be8053d4f89bd3d075a4cbf6"} Sep 06 06:26:45.268419 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.268386 2580 scope.go:117] "RemoveContainer" containerID="31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2" Sep 06 06:26:45.278162 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.278144 2580 scope.go:117] "RemoveContainer" containerID="31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2" Sep 06 06:26:45.278416 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:26:45.278398 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2\": container with ID starting with 31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2 not found: ID does not exist" containerID="31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2" Sep 06 06:26:45.278486 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.278421 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2"} err="failed to get container status \"31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2\": rpc error: code = NotFound desc = could not find container \"31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2\": container with ID starting with 31bc1860e6fdaaa0313d1bb6a7d4305958901275a34ecd02935ddccedb7efda2 not found: ID does not exist" Sep 06 06:26:45.291856 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.291810 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp"] Sep 06 06:26:45.293740 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:45.293700 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a36847/ipp-route-a36847-kserve-54885979ff-6p8tp"] Sep 06 06:26:46.875533 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:26:46.875497 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baa44881-b235-4e82-869f-977eab080e46" path="/var/lib/kubelet/pods/baa44881-b235-4e82-869f-977eab080e46/volumes" Sep 06 06:27:27.095000 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.094962 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx"] Sep 06 06:27:27.095499 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.095387 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c28c5c-8d17-43f0-abfc-0c0a82609034" containerName="revoke-keys" Sep 06 06:27:27.095499 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.095400 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c28c5c-8d17-43f0-abfc-0c0a82609034" containerName="revoke-keys" Sep 06 06:27:27.095499 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.095420 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a566d5-c36b-4a30-ac38-077588601917" containerName="payload-pre-processing" Sep 06 06:27:27.095499 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.095426 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a566d5-c36b-4a30-ac38-077588601917" containerName="payload-pre-processing" Sep 06 06:27:27.095499 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.095434 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa44881-b235-4e82-869f-977eab080e46" containerName="main" Sep 06 06:27:27.095499 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.095439 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa44881-b235-4e82-869f-977eab080e46" containerName="main" Sep 06 06:27:27.095499 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.095503 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="baa44881-b235-4e82-869f-977eab080e46" containerName="main" Sep 06 06:27:27.095726 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.095511 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c28c5c-8d17-43f0-abfc-0c0a82609034" containerName="revoke-keys" Sep 06 06:27:27.095726 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.095520 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a566d5-c36b-4a30-ac38-077588601917" containerName="payload-pre-processing" Sep 06 06:27:27.098785 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.098768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx" Sep 06 06:27:27.101302 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.101280 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:27:27.101425 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.101360 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:27:27.106846 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.106807 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx"] Sep 06 06:27:27.120132 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.120107 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8376755-f204-4def-be9d-16f9be408c2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx\" (UID: \"f8376755-f204-4def-be9d-16f9be408c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx" Sep 06 06:27:27.220674 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.220639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8376755-f204-4def-be9d-16f9be408c2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx\" (UID: \"f8376755-f204-4def-be9d-16f9be408c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx" Sep 06 06:27:27.221311 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.221288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8376755-f204-4def-be9d-16f9be408c2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx\" (UID: \"f8376755-f204-4def-be9d-16f9be408c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx" Sep 06 06:27:27.410733 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.410646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx" Sep 06 06:27:27.679556 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:27.679532 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx"] Sep 06 06:27:27.681843 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:27:27.681806 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8376755_f204_4def_be9d_16f9be408c2e.slice/crio-c200f88d6cde30e07bebc68ebcbf3d7969c5d88080cc00400ac5952ed6408607 WatchSource:0}: Error finding container c200f88d6cde30e07bebc68ebcbf3d7969c5d88080cc00400ac5952ed6408607: Status 404 returned error can't find the container with id c200f88d6cde30e07bebc68ebcbf3d7969c5d88080cc00400ac5952ed6408607 Sep 06 06:27:28.487805 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:28.487769 2580 generic.go:356] "Generic (PLEG): container finished" podID="f8376755-f204-4def-be9d-16f9be408c2e" containerID="41274c9fad0b94d2e0aacb825ff99e3305b09bf1bd2972430108e69dcdcb523f" exitCode=0 Sep 06 06:27:28.488239 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:28.487857 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx" event={"ID":"f8376755-f204-4def-be9d-16f9be408c2e","Type":"ContainerDied","Data":"41274c9fad0b94d2e0aacb825ff99e3305b09bf1bd2972430108e69dcdcb523f"} Sep 06 06:27:28.488239 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:28.487935 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx" event={"ID":"f8376755-f204-4def-be9d-16f9be408c2e","Type":"ContainerStarted","Data":"c200f88d6cde30e07bebc68ebcbf3d7969c5d88080cc00400ac5952ed6408607"} Sep 06 06:27:29.622633 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:29.622609 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx" Sep 06 06:27:29.637241 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:29.637211 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8376755-f204-4def-be9d-16f9be408c2e-maas-api-service-ca\") pod \"f8376755-f204-4def-be9d-16f9be408c2e\" (UID: \"f8376755-f204-4def-be9d-16f9be408c2e\") " Sep 06 06:27:29.637590 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:29.637559 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8376755-f204-4def-be9d-16f9be408c2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8376755-f204-4def-be9d-16f9be408c2e" (UID: "f8376755-f204-4def-be9d-16f9be408c2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:27:29.737983 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:29.737946 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8376755-f204-4def-be9d-16f9be408c2e-maas-api-service-ca\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:27:30.499161 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:30.499124 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx" Sep 06 06:27:30.499161 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:30.499146 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx" event={"ID":"f8376755-f204-4def-be9d-16f9be408c2e","Type":"ContainerDied","Data":"c200f88d6cde30e07bebc68ebcbf3d7969c5d88080cc00400ac5952ed6408607"} Sep 06 06:27:30.499373 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:30.499198 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c200f88d6cde30e07bebc68ebcbf3d7969c5d88080cc00400ac5952ed6408607" Sep 06 06:27:37.560013 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:37.559962 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9"] Sep 06 06:27:37.561061 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:37.561024 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" podUID="42a18e35-4ca4-4784-ba22-873e5272e440" containerName="payload-pre-processing" containerID="cri-o://63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32" gracePeriod=30 Sep 06 06:27:37.715100 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:37.715065 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx"] Sep 06 06:27:37.717417 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:37.717395 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a52b50-m4phx"] Sep 06 06:27:37.811354 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:37.811291 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" Sep 06 06:27:37.908335 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:37.908306 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a18e35-4ca4-4784-ba22-873e5272e440-plugins-config-volume\") pod \"42a18e35-4ca4-4784-ba22-873e5272e440\" (UID: \"42a18e35-4ca4-4784-ba22-873e5272e440\") " Sep 06 06:27:37.908476 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:37.908352 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-962xd\" (UniqueName: \"kubernetes.io/projected/42a18e35-4ca4-4784-ba22-873e5272e440-kube-api-access-962xd\") pod \"42a18e35-4ca4-4784-ba22-873e5272e440\" (UID: \"42a18e35-4ca4-4784-ba22-873e5272e440\") " Sep 06 06:27:37.908657 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:37.908634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a18e35-4ca4-4784-ba22-873e5272e440-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42a18e35-4ca4-4784-ba22-873e5272e440" (UID: "42a18e35-4ca4-4784-ba22-873e5272e440"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:27:37.910473 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:37.910449 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a18e35-4ca4-4784-ba22-873e5272e440-kube-api-access-962xd" (OuterVolumeSpecName: "kube-api-access-962xd") pod "42a18e35-4ca4-4784-ba22-873e5272e440" (UID: "42a18e35-4ca4-4784-ba22-873e5272e440"). InnerVolumeSpecName "kube-api-access-962xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:27:38.009538 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.009508 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a18e35-4ca4-4784-ba22-873e5272e440-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:27:38.009538 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.009533 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-962xd\" (UniqueName: \"kubernetes.io/projected/42a18e35-4ca4-4784-ba22-873e5272e440-kube-api-access-962xd\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:27:38.541516 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.541477 2580 generic.go:356] "Generic (PLEG): container finished" podID="42a18e35-4ca4-4784-ba22-873e5272e440" containerID="63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32" exitCode=0 Sep 06 06:27:38.541690 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.541526 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" Sep 06 06:27:38.541690 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.541526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" event={"ID":"42a18e35-4ca4-4784-ba22-873e5272e440","Type":"ContainerDied","Data":"63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32"} Sep 06 06:27:38.541690 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.541579 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9" event={"ID":"42a18e35-4ca4-4784-ba22-873e5272e440","Type":"ContainerDied","Data":"0b108db18f3196c5f4aed2de6b6b891d3b17a4e02545b57af22cfb2865574969"} Sep 06 06:27:38.541690 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.541610 2580 scope.go:117] "RemoveContainer" containerID="63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32" Sep 06 06:27:38.550489 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.550469 2580 scope.go:117] "RemoveContainer" containerID="63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32" Sep 06 06:27:38.550766 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:27:38.550745 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32\": container with ID starting with 63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32 not found: ID does not exist" containerID="63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32" Sep 06 06:27:38.550845 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.550777 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32"} err="failed to get container status \"63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32\": rpc error: code = NotFound desc = could not find container \"63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32\": container with ID starting with 63fc91a2b715b990a92eb3cf2a038898e325281f5523baa670d53ec4b91b9e32 not found: ID does not exist" Sep 06 06:27:38.565622 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.565594 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9"] Sep 06 06:27:38.569599 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.569578 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a52b50-7c8c7b6d74-7z6s9"] Sep 06 06:27:38.875923 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.875807 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a18e35-4ca4-4784-ba22-873e5272e440" path="/var/lib/kubelet/pods/42a18e35-4ca4-4784-ba22-873e5272e440/volumes" Sep 06 06:27:38.876280 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:38.876259 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8376755-f204-4def-be9d-16f9be408c2e" path="/var/lib/kubelet/pods/f8376755-f204-4def-be9d-16f9be408c2e/volumes" Sep 06 06:27:55.140668 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:27:55.140638 2580 scope.go:117] "RemoveContainer" containerID="432fd128ae2b1574f86540b250c8dc2bef03e5f51f0b20a21f3dfca45806e17f" Sep 06 06:28:12.452721 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.452649 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx"] Sep 06 06:28:12.453376 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.452971 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" podUID="85a9d189-2ffa-4d9e-bfac-a29c14508d64" containerName="payload-processing" containerID="cri-o://b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc" gracePeriod=30 Sep 06 06:28:12.456914 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.456853 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g"] Sep 06 06:28:12.457345 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.457297 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" podUID="84108c7d-4f79-4ec5-bd64-93cb13d1e051" containerName="payload-pre-processing" containerID="cri-o://0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a" gracePeriod=30 Sep 06 06:28:12.716646 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.716587 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" Sep 06 06:28:12.737645 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.735056 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" Sep 06 06:28:12.740696 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.740662 2580 generic.go:356] "Generic (PLEG): container finished" podID="85a9d189-2ffa-4d9e-bfac-a29c14508d64" containerID="b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc" exitCode=0 Sep 06 06:28:12.740867 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.740708 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" Sep 06 06:28:12.740867 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.740745 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" event={"ID":"85a9d189-2ffa-4d9e-bfac-a29c14508d64","Type":"ContainerDied","Data":"b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc"} Sep 06 06:28:12.740867 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.740805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx" event={"ID":"85a9d189-2ffa-4d9e-bfac-a29c14508d64","Type":"ContainerDied","Data":"18d66f664f1a8dcc3915df60404fa5656c4cc85a607a029a801d5468a40c2c4e"} Sep 06 06:28:12.740867 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.740858 2580 scope.go:117] "RemoveContainer" containerID="b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc" Sep 06 06:28:12.742476 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.742453 2580 generic.go:356] "Generic (PLEG): container finished" podID="84108c7d-4f79-4ec5-bd64-93cb13d1e051" containerID="0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a" exitCode=0 Sep 06 06:28:12.742556 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.742482 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" Sep 06 06:28:12.742556 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.742530 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" event={"ID":"84108c7d-4f79-4ec5-bd64-93cb13d1e051","Type":"ContainerDied","Data":"0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a"} Sep 06 06:28:12.742647 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.742578 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g" event={"ID":"84108c7d-4f79-4ec5-bd64-93cb13d1e051","Type":"ContainerDied","Data":"a526f096e511862659eed253cc27af3c28f9f10f32872f71174dd6be2b6adade"} Sep 06 06:28:12.750100 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.750071 2580 scope.go:117] "RemoveContainer" containerID="b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc" Sep 06 06:28:12.750467 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:28:12.750445 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc\": container with ID starting with b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc not found: ID does not exist" containerID="b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc" Sep 06 06:28:12.750553 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.750479 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc"} err="failed to get container status \"b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc\": rpc error: code = NotFound desc = could not find container \"b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc\": container with ID starting with b36dfd1d85e52d1c8ccb5ee0b05ae1ba4a619d184c43256376391099cdd396cc not found: ID does not exist" Sep 06 06:28:12.750553 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.750502 2580 scope.go:117] "RemoveContainer" containerID="0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a" Sep 06 06:28:12.760035 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.760013 2580 scope.go:117] "RemoveContainer" containerID="0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a" Sep 06 06:28:12.760312 ip-10-0-142-34 kubenswrapper[2580]: E0906 06:28:12.760292 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a\": container with ID starting with 0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a not found: ID does not exist" containerID="0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a" Sep 06 06:28:12.760388 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.760323 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a"} err="failed to get container status \"0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a\": rpc error: code = NotFound desc = could not find container \"0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a\": container with ID starting with 0045798fbb0700c3b4e2373f431b0a0a2ca8b67ee81c4064a3d359febb36022a not found: ID does not exist" Sep 06 06:28:12.813468 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.813436 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84108c7d-4f79-4ec5-bd64-93cb13d1e051-plugins-config-volume\") pod \"84108c7d-4f79-4ec5-bd64-93cb13d1e051\" (UID: \"84108c7d-4f79-4ec5-bd64-93cb13d1e051\") " Sep 06 06:28:12.813468 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.813471 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a9d189-2ffa-4d9e-bfac-a29c14508d64-plugins-config-volume\") pod \"85a9d189-2ffa-4d9e-bfac-a29c14508d64\" (UID: \"85a9d189-2ffa-4d9e-bfac-a29c14508d64\") " Sep 06 06:28:12.813699 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.813595 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f8sc\" (UniqueName: \"kubernetes.io/projected/85a9d189-2ffa-4d9e-bfac-a29c14508d64-kube-api-access-9f8sc\") pod \"85a9d189-2ffa-4d9e-bfac-a29c14508d64\" (UID: \"85a9d189-2ffa-4d9e-bfac-a29c14508d64\") " Sep 06 06:28:12.813699 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.813623 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzcv2\" (UniqueName: \"kubernetes.io/projected/84108c7d-4f79-4ec5-bd64-93cb13d1e051-kube-api-access-pzcv2\") pod \"84108c7d-4f79-4ec5-bd64-93cb13d1e051\" (UID: \"84108c7d-4f79-4ec5-bd64-93cb13d1e051\") " Sep 06 06:28:12.813926 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.813900 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84108c7d-4f79-4ec5-bd64-93cb13d1e051-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84108c7d-4f79-4ec5-bd64-93cb13d1e051" (UID: "84108c7d-4f79-4ec5-bd64-93cb13d1e051"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:28:12.813990 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.813907 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a9d189-2ffa-4d9e-bfac-a29c14508d64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85a9d189-2ffa-4d9e-bfac-a29c14508d64" (UID: "85a9d189-2ffa-4d9e-bfac-a29c14508d64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:28:12.815676 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.815650 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84108c7d-4f79-4ec5-bd64-93cb13d1e051-kube-api-access-pzcv2" (OuterVolumeSpecName: "kube-api-access-pzcv2") pod "84108c7d-4f79-4ec5-bd64-93cb13d1e051" (UID: "84108c7d-4f79-4ec5-bd64-93cb13d1e051"). InnerVolumeSpecName "kube-api-access-pzcv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:28:12.815737 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.815659 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a9d189-2ffa-4d9e-bfac-a29c14508d64-kube-api-access-9f8sc" (OuterVolumeSpecName: "kube-api-access-9f8sc") pod "85a9d189-2ffa-4d9e-bfac-a29c14508d64" (UID: "85a9d189-2ffa-4d9e-bfac-a29c14508d64"). InnerVolumeSpecName "kube-api-access-9f8sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:28:12.915185 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.915151 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84108c7d-4f79-4ec5-bd64-93cb13d1e051-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:28:12.915185 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.915179 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a9d189-2ffa-4d9e-bfac-a29c14508d64-plugins-config-volume\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:28:12.915348 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.915193 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f8sc\" (UniqueName: \"kubernetes.io/projected/85a9d189-2ffa-4d9e-bfac-a29c14508d64-kube-api-access-9f8sc\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:28:12.915348 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:12.915205 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzcv2\" (UniqueName: \"kubernetes.io/projected/84108c7d-4f79-4ec5-bd64-93cb13d1e051-kube-api-access-pzcv2\") on node \"ip-10-0-142-34.ec2.internal\" DevicePath \"\"" Sep 06 06:28:13.058436 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:13.058362 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx"] Sep 06 06:28:13.062135 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:13.062103 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dc97f9-5f9f56d66f-hnqmx"] Sep 06 06:28:13.072436 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:13.072411 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g"] Sep 06 06:28:13.075787 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:13.075767 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dc97f9-56754b8f4b-7kw8g"] Sep 06 06:28:14.875617 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:14.875582 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84108c7d-4f79-4ec5-bd64-93cb13d1e051" path="/var/lib/kubelet/pods/84108c7d-4f79-4ec5-bd64-93cb13d1e051/volumes" Sep 06 06:28:14.876005 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:14.875946 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85a9d189-2ffa-4d9e-bfac-a29c14508d64" path="/var/lib/kubelet/pods/85a9d189-2ffa-4d9e-bfac-a29c14508d64/volumes" Sep 06 06:28:55.164444 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:28:55.164411 2580 scope.go:117] "RemoveContainer" containerID="1cce6554b9b68a3ee1626f85a41082870c76721d81b895c6d1d6ccdc6ebe4f38" Sep 06 06:30:00.129448 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129365 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811270-lcqs5"] Sep 06 06:30:00.129862 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129754 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84108c7d-4f79-4ec5-bd64-93cb13d1e051" containerName="payload-pre-processing" Sep 06 06:30:00.129862 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129764 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84108c7d-4f79-4ec5-bd64-93cb13d1e051" containerName="payload-pre-processing" Sep 06 06:30:00.129862 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129773 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8376755-f204-4def-be9d-16f9be408c2e" containerName="revoke-keys" Sep 06 06:30:00.129862 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129778 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8376755-f204-4def-be9d-16f9be408c2e" containerName="revoke-keys" Sep 06 06:30:00.129862 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129791 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a9d189-2ffa-4d9e-bfac-a29c14508d64" containerName="payload-processing" Sep 06 06:30:00.129862 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129797 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a9d189-2ffa-4d9e-bfac-a29c14508d64" containerName="payload-processing" Sep 06 06:30:00.129862 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129812 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a18e35-4ca4-4784-ba22-873e5272e440" containerName="payload-pre-processing" Sep 06 06:30:00.129862 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129818 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a18e35-4ca4-4784-ba22-873e5272e440" containerName="payload-pre-processing" Sep 06 06:30:00.130112 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129904 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="84108c7d-4f79-4ec5-bd64-93cb13d1e051" containerName="payload-pre-processing" Sep 06 06:30:00.130112 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129912 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a18e35-4ca4-4784-ba22-873e5272e440" containerName="payload-pre-processing" Sep 06 06:30:00.130112 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129920 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8376755-f204-4def-be9d-16f9be408c2e" containerName="revoke-keys" Sep 06 06:30:00.130112 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.129926 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a9d189-2ffa-4d9e-bfac-a29c14508d64" containerName="payload-processing" Sep 06 06:30:00.133026 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.133011 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811270-lcqs5" Sep 06 06:30:00.135340 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.135317 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:30:00.139304 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.139280 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811270-lcqs5"] Sep 06 06:30:00.143994 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.143975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811270-lcqs5" Sep 06 06:30:00.412379 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:00.412355 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811270-lcqs5"] Sep 06 06:30:00.414366 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:30:00.414337 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ee8e79_d5e0_4626_a920_94e8d9b31bb7.slice/crio-975538d0f3ca8c9ad2a433cfa99c5a1e3c3dc6e85255cdd0a114bbcb08cc8cad WatchSource:0}: Error finding container 975538d0f3ca8c9ad2a433cfa99c5a1e3c3dc6e85255cdd0a114bbcb08cc8cad: Status 404 returned error can't find the container with id 975538d0f3ca8c9ad2a433cfa99c5a1e3c3dc6e85255cdd0a114bbcb08cc8cad Sep 06 06:30:01.293533 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:01.293499 2580 generic.go:356] "Generic (PLEG): container finished" podID="64ee8e79-d5e0-4626-a920-94e8d9b31bb7" containerID="31caff453c764d4730b689ae7e060420634fa78f3d6d4d4164e55b02f1406060" exitCode=0 Sep 06 06:30:01.293954 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:01.293581 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811270-lcqs5" event={"ID":"64ee8e79-d5e0-4626-a920-94e8d9b31bb7","Type":"ContainerDied","Data":"31caff453c764d4730b689ae7e060420634fa78f3d6d4d4164e55b02f1406060"} Sep 06 06:30:01.293954 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:01.293630 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811270-lcqs5" event={"ID":"64ee8e79-d5e0-4626-a920-94e8d9b31bb7","Type":"ContainerStarted","Data":"975538d0f3ca8c9ad2a433cfa99c5a1e3c3dc6e85255cdd0a114bbcb08cc8cad"} Sep 06 06:30:02.426445 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:02.426417 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811270-lcqs5" Sep 06 06:30:03.304773 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:03.304731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811270-lcqs5" event={"ID":"64ee8e79-d5e0-4626-a920-94e8d9b31bb7","Type":"ContainerDied","Data":"975538d0f3ca8c9ad2a433cfa99c5a1e3c3dc6e85255cdd0a114bbcb08cc8cad"} Sep 06 06:30:03.304773 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:03.304759 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811270-lcqs5" Sep 06 06:30:03.304982 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:30:03.304790 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="975538d0f3ca8c9ad2a433cfa99c5a1e3c3dc6e85255cdd0a114bbcb08cc8cad" Sep 06 06:31:55.205341 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:31:55.205311 2580 scope.go:117] "RemoveContainer" containerID="593946e85d57ab4daa1e4274165bb06f99e03ff4148664fb1451a6e14acdb19c" Sep 06 06:32:55.224440 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:32:55.224369 2580 scope.go:117] "RemoveContainer" containerID="ddbd4f6f80a9fae2aa6594b70524bdd916bdc5ad8c1c33ddf6ba753c9046275f" Sep 06 06:33:55.243168 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:33:55.243137 2580 scope.go:117] "RemoveContainer" containerID="41274c9fad0b94d2e0aacb825ff99e3305b09bf1bd2972430108e69dcdcb523f" Sep 06 06:34:15.252530 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:15.252489 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d6d57f484-zh75v_f3caa196-731c-4c20-85ea-c52c1d4a465a/authorino/0.log" Sep 06 06:34:21.024990 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:21.024924 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b897cd959-76mrw_1def8cda-27d7-4d4b-9f76-85b083cad401/copy-manifests/0.log" Sep 06 06:34:21.030715 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:21.030691 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b897cd959-76mrw_1def8cda-27d7-4d4b-9f76-85b083cad401/manager/0.log" Sep 06 06:34:21.919480 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:21.919450 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29811270-lcqs5_64ee8e79-d5e0-4626-a920-94e8d9b31bb7/cleanup/0.log" Sep 06 06:34:22.778241 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:22.778214 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d6d57f484-zh75v_f3caa196-731c-4c20-85ea-c52c1d4a465a/authorino/0.log" Sep 06 06:34:22.891226 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:22.891189 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-5gnds_e4393b5c-d446-4dbc-8070-b12a57dd98a9/manager/0.log" Sep 06 06:34:23.000148 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:23.000116 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-n875r_dcf33542-3d05-4fb3-a4e8-76ac2142fdd7/manager/0.log" Sep 06 06:34:24.113708 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:24.113678 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-696f56895b-nklb9_b3da34e8-2b57-4e9d-8f2a-fd7067b2c590/kube-auth-proxy/0.log" Sep 06 06:34:24.455562 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:24.455535 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-95mv6_287f1091-aa00-4497-badd-63c872f52fd6/payload-processing/0.log" Sep 06 06:34:25.118698 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:25.118665 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-r5n2x_466a57ef-825b-41d2-9355-e1cc6cea3d03/main/0.log" Sep 06 06:34:25.232045 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:25.232017 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-thlff_9966d16e-844e-4b92-928b-e110d04c0426/main/0.log" Sep 06 06:34:25.454065 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:25.454032 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ww76c_22be0ec4-6100-49d7-a2c2-5caac18d8d14/main/0.log" Sep 06 06:34:25.561556 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:25.561528 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-nvs9b_54fa6afd-4651-429c-adb9-1858950002bc/main/0.log" Sep 06 06:34:34.007399 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:34.007366 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nksgl_8e95c4ec-ee79-4651-9f84-6f1bd6660624/global-pull-secret-syncer/0.log" Sep 06 06:34:34.149118 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:34.149085 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dmd4j_81e0c9ba-7535-45d5-933f-4aa8c33194c0/konnectivity-agent/0.log" Sep 06 06:34:34.217748 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:34.217712 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-34.ec2.internal_6b5baafa68872d0e1a89c3b7fe397e3a/haproxy/0.log" Sep 06 06:34:39.429850 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:39.429799 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d6d57f484-zh75v_f3caa196-731c-4c20-85ea-c52c1d4a465a/authorino/0.log" Sep 06 06:34:39.461574 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:39.461538 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-5gnds_e4393b5c-d446-4dbc-8070-b12a57dd98a9/manager/0.log" Sep 06 06:34:39.486047 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:39.486019 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-n875r_dcf33542-3d05-4fb3-a4e8-76ac2142fdd7/manager/0.log" Sep 06 06:34:41.112809 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.112727 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-bh6dx_b1f02946-f574-44dd-be9e-bb9fbaef6be6/cluster-monitoring-operator/0.log" Sep 06 06:34:41.144538 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.144509 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-qq47v_c5e8bcb1-6f79-4e6c-b866-94163a95fd7f/kube-state-metrics/0.log" Sep 06 06:34:41.168030 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.167994 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-qq47v_c5e8bcb1-6f79-4e6c-b866-94163a95fd7f/kube-rbac-proxy-main/0.log" Sep 06 06:34:41.189330 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.189300 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-qq47v_c5e8bcb1-6f79-4e6c-b866-94163a95fd7f/kube-rbac-proxy-self/0.log" Sep 06 06:34:41.222471 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.222441 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5ffd6c4df4-xjbbz_5b7403db-7ece-4d47-b995-cd4c4a139c09/metrics-server/0.log" Sep 06 06:34:41.367849 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.367756 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9hfg6_fd6e43a0-e309-4afd-855b-f9ac0d5cec2b/node-exporter/0.log" Sep 06 06:34:41.387569 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.387544 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9hfg6_fd6e43a0-e309-4afd-855b-f9ac0d5cec2b/kube-rbac-proxy/0.log" Sep 06 06:34:41.408734 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.408709 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9hfg6_fd6e43a0-e309-4afd-855b-f9ac0d5cec2b/init-textfile/0.log" Sep 06 06:34:41.521521 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.521481 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-czz49_33ed7158-81bb-42b5-9afa-7e6cbf9e278a/kube-rbac-proxy-main/0.log" Sep 06 06:34:41.545819 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.545790 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-czz49_33ed7158-81bb-42b5-9afa-7e6cbf9e278a/kube-rbac-proxy-self/0.log" Sep 06 06:34:41.571452 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.571422 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-czz49_33ed7158-81bb-42b5-9afa-7e6cbf9e278a/openshift-state-metrics/0.log" Sep 06 06:34:41.610025 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.609991 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_253382d3-9a00-4927-8088-358f76a673b6/prometheus/0.log" Sep 06 06:34:41.633924 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.633812 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_253382d3-9a00-4927-8088-358f76a673b6/config-reloader/0.log" Sep 06 06:34:41.654031 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.653993 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_253382d3-9a00-4927-8088-358f76a673b6/thanos-sidecar/0.log" Sep 06 06:34:41.674855 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.674801 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_253382d3-9a00-4927-8088-358f76a673b6/kube-rbac-proxy-web/0.log" Sep 06 06:34:41.696560 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.696486 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_253382d3-9a00-4927-8088-358f76a673b6/kube-rbac-proxy/0.log" Sep 06 06:34:41.718506 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.718472 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_253382d3-9a00-4927-8088-358f76a673b6/kube-rbac-proxy-thanos/0.log" Sep 06 06:34:41.741376 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.741348 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_253382d3-9a00-4927-8088-358f76a673b6/init-config-reloader/0.log" Sep 06 06:34:41.848246 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.848213 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-tsvv6_e171e326-abfb-452f-929c-14d0d29e9ffa/prometheus-operator-admission-webhook/0.log" Sep 06 06:34:41.980452 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:41.980415 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cb56fbf6c-4n8qr_2ed6c30a-5c06-4d62-9a24-b5f17515ddf6/thanos-query/0.log" Sep 06 06:34:42.004051 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:42.004015 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cb56fbf6c-4n8qr_2ed6c30a-5c06-4d62-9a24-b5f17515ddf6/kube-rbac-proxy-web/0.log" Sep 06 06:34:42.030747 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:42.030717 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cb56fbf6c-4n8qr_2ed6c30a-5c06-4d62-9a24-b5f17515ddf6/kube-rbac-proxy/0.log" Sep 06 06:34:42.061959 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:42.061928 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cb56fbf6c-4n8qr_2ed6c30a-5c06-4d62-9a24-b5f17515ddf6/prom-label-proxy/0.log" Sep 06 06:34:42.084985 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:42.084953 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cb56fbf6c-4n8qr_2ed6c30a-5c06-4d62-9a24-b5f17515ddf6/kube-rbac-proxy-rules/0.log" Sep 06 06:34:42.112064 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:42.112037 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cb56fbf6c-4n8qr_2ed6c30a-5c06-4d62-9a24-b5f17515ddf6/kube-rbac-proxy-metrics/0.log" Sep 06 06:34:43.559249 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:43.559215 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-f7dzz_d9b61ebb-d415-4305-abd5-d712c4d0ced3/networking-console-plugin/0.log" Sep 06 06:34:44.695353 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.695320 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz"] Sep 06 06:34:44.695990 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.695968 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ee8e79-d5e0-4626-a920-94e8d9b31bb7" containerName="cleanup" Sep 06 06:34:44.695990 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.695991 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ee8e79-d5e0-4626-a920-94e8d9b31bb7" containerName="cleanup" Sep 06 06:34:44.696171 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.696079 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ee8e79-d5e0-4626-a920-94e8d9b31bb7" containerName="cleanup" Sep 06 06:34:44.699318 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.699302 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.701791 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.701769 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zzhm6\"/\"default-dockercfg-gv2n8\"" Sep 06 06:34:44.703072 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.703054 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzhm6\"/\"openshift-service-ca.crt\"" Sep 06 06:34:44.703172 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.703054 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzhm6\"/\"kube-root-ca.crt\"" Sep 06 06:34:44.709215 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.709188 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz"] Sep 06 06:34:44.789414 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.789378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6eb821fd-591b-4c83-bdbd-f49e819b7240-sys\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.789598 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.789426 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6eb821fd-591b-4c83-bdbd-f49e819b7240-lib-modules\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.789598 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.789490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6eb821fd-591b-4c83-bdbd-f49e819b7240-proc\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.789598 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.789554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvttl\" (UniqueName: \"kubernetes.io/projected/6eb821fd-591b-4c83-bdbd-f49e819b7240-kube-api-access-dvttl\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.789598 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.789595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6eb821fd-591b-4c83-bdbd-f49e819b7240-podres\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.890578 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.890544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6eb821fd-591b-4c83-bdbd-f49e819b7240-proc\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.890767 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.890620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvttl\" (UniqueName: \"kubernetes.io/projected/6eb821fd-591b-4c83-bdbd-f49e819b7240-kube-api-access-dvttl\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.890767 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.890653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6eb821fd-591b-4c83-bdbd-f49e819b7240-podres\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.890767 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.890670 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6eb821fd-591b-4c83-bdbd-f49e819b7240-proc\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.890767 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.890677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6eb821fd-591b-4c83-bdbd-f49e819b7240-sys\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.890767 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.890758 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6eb821fd-591b-4c83-bdbd-f49e819b7240-sys\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.890767 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.890764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6eb821fd-591b-4c83-bdbd-f49e819b7240-lib-modules\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.891082 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.890780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6eb821fd-591b-4c83-bdbd-f49e819b7240-podres\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.891082 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.890917 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6eb821fd-591b-4c83-bdbd-f49e819b7240-lib-modules\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:44.898883 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:44.898861 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvttl\" (UniqueName: \"kubernetes.io/projected/6eb821fd-591b-4c83-bdbd-f49e819b7240-kube-api-access-dvttl\") pod \"perf-node-gather-daemonset-7pzfz\" (UID: \"6eb821fd-591b-4c83-bdbd-f49e819b7240\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:45.011212 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:45.011100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:45.281100 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:45.281065 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz"] Sep 06 06:34:45.283800 ip-10-0-142-34 kubenswrapper[2580]: W0906 06:34:45.283767 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6eb821fd_591b_4c83_bdbd_f49e819b7240.slice/crio-5814c2fc406522cb18ef5889103d859990f1a24cd3290a2251fac7b57c7c3325 WatchSource:0}: Error finding container 5814c2fc406522cb18ef5889103d859990f1a24cd3290a2251fac7b57c7c3325: Status 404 returned error can't find the container with id 5814c2fc406522cb18ef5889103d859990f1a24cd3290a2251fac7b57c7c3325 Sep 06 06:34:45.440416 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:45.440380 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-r5h5s_eb6157db-5f6c-49a2-afd6-732fae965430/volume-data-source-validator/0.log" Sep 06 06:34:45.704630 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:45.704591 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" event={"ID":"6eb821fd-591b-4c83-bdbd-f49e819b7240","Type":"ContainerStarted","Data":"26dc29e9f57812ca2d98176132312cd42a27b437b938ff3b1ba17373a4e57c71"} Sep 06 06:34:45.705098 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:45.704647 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" event={"ID":"6eb821fd-591b-4c83-bdbd-f49e819b7240","Type":"ContainerStarted","Data":"5814c2fc406522cb18ef5889103d859990f1a24cd3290a2251fac7b57c7c3325"} Sep 06 06:34:45.705098 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:45.704700 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:45.720171 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:45.720117 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" podStartSLOduration=1.720100171 podStartE2EDuration="1.720100171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 06:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:34:45.718522787 +0000 UTC m=+2451.367141966" watchObservedRunningTime="2026-09-06 06:34:45.720100171 +0000 UTC m=+2451.368719350" Sep 06 06:34:46.500125 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:46.500083 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5g9s6_1ecd3697-4253-4467-a57b-368c0950ffe2/dns/0.log" Sep 06 06:34:46.519810 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:46.519778 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5g9s6_1ecd3697-4253-4467-a57b-368c0950ffe2/kube-rbac-proxy/0.log" Sep 06 06:34:46.685541 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:46.685513 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xxk4z_27b14080-015b-496f-b375-cc4cbdbf1892/dns-node-resolver/0.log" Sep 06 06:34:47.270096 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:47.270065 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cb5np_ce82693f-a593-41cf-83c0-d09f07d50168/node-ca/0.log" Sep 06 06:34:48.514192 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:48.514165 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-696f56895b-nklb9_b3da34e8-2b57-4e9d-8f2a-fd7067b2c590/kube-auth-proxy/0.log" Sep 06 06:34:48.625430 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:48.625393 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-95mv6_287f1091-aa00-4497-badd-63c872f52fd6/payload-processing/0.log" Sep 06 06:34:49.557844 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:49.557797 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9r8b_4354b120-a2f7-43f8-b3e7-be55e37d543f/serve-healthcheck-canary/0.log" Sep 06 06:34:50.181011 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:50.180976 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tzmc6_627d4abe-9c46-41be-9d8b-ae68e88ce293/kube-rbac-proxy/0.log" Sep 06 06:34:50.200999 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:50.200967 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tzmc6_627d4abe-9c46-41be-9d8b-ae68e88ce293/exporter/0.log" Sep 06 06:34:50.221572 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:50.221535 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tzmc6_627d4abe-9c46-41be-9d8b-ae68e88ce293/extractor/0.log" Sep 06 06:34:51.720559 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:51.720533 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-7pzfz" Sep 06 06:34:52.743813 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:52.743786 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b897cd959-76mrw_1def8cda-27d7-4d4b-9f76-85b083cad401/manager/0.log" Sep 06 06:34:52.764173 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:52.764152 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b897cd959-76mrw_1def8cda-27d7-4d4b-9f76-85b083cad401/copy-manifests/0.log" Sep 06 06:34:54.445942 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:54.445907 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6fb67bfd69-hhxwr_99f89eb3-6fcb-4940-8d39-a2102c52da51/manager/0.log" Sep 06 06:34:55.515394 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:55.515362 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-696f56895b-nklb9_b3da34e8-2b57-4e9d-8f2a-fd7067b2c590/kube-auth-proxy/0.log" Sep 06 06:34:55.593541 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:55.593507 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-95mv6_287f1091-aa00-4497-badd-63c872f52fd6/payload-processing/0.log" Sep 06 06:34:57.504570 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:57.504540 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-696f56895b-nklb9_b3da34e8-2b57-4e9d-8f2a-fd7067b2c590/kube-auth-proxy/0.log" Sep 06 06:34:57.598507 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:57.598472 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-95mv6_287f1091-aa00-4497-badd-63c872f52fd6/payload-processing/0.log" Sep 06 06:34:59.756153 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:59.756122 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-zwkhp_721c23a4-d6f1-4f87-836e-741f52dced36/migrator/0.log" Sep 06 06:34:59.774986 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:34:59.774952 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-zwkhp_721c23a4-d6f1-4f87-836e-741f52dced36/graceful-termination/0.log" Sep 06 06:35:01.232379 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:35:01.232347 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqfs6_678e8aa3-c088-4f39-bf3d-f158f45c9aef/kube-multus-additional-cni-plugins/0.log" Sep 06 06:35:01.252218 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:35:01.252188 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqfs6_678e8aa3-c088-4f39-bf3d-f158f45c9aef/egress-router-binary-copy/0.log" Sep 06 06:35:01.274646 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:35:01.274619 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqfs6_678e8aa3-c088-4f39-bf3d-f158f45c9aef/cni-plugins/0.log" Sep 06 06:35:01.294297 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:35:01.294272 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqfs6_678e8aa3-c088-4f39-bf3d-f158f45c9aef/bond-cni-plugin/0.log" Sep 06 06:35:01.314443 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:35:01.314395 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqfs6_678e8aa3-c088-4f39-bf3d-f158f45c9aef/routeoverride-cni/0.log" Sep 06 06:35:01.333737 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:35:01.333714 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqfs6_678e8aa3-c088-4f39-bf3d-f158f45c9aef/whereabouts-cni-bincopy/0.log" Sep 06 06:35:01.352983 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:35:01.352962 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cqfs6_678e8aa3-c088-4f39-bf3d-f158f45c9aef/whereabouts-cni/0.log" Sep 06 06:35:01.630810 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:35:01.630727 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gjhdl_513bdbba-c5b6-4ea9-80ea-f909cbba8abb/kube-multus/0.log" Sep 06 06:35:01.652196 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:35:01.652168 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2z6qh_e5ada9e9-c4df-486d-9325-b5dcecdd451b/network-metrics-daemon/0.log" Sep 06 06:35:01.671835 ip-10-0-142-34 kubenswrapper[2580]: I0906 06:35:01.671796 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2z6qh_e5ada9e9-c4df-486d-9325-b5dcecdd451b/kube-rbac-proxy/0.log"